Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for storing data in a version control system. One of the methods includes maintaining, in a data store, a tree-structured index of files in which each leaf node stores an entry for each file in a plurality of files for a snapshot that includes a unique file identifier for the respective file; receiving a request for a particular file; generating a hash of a particular file path for the particular file; identifying, using the hash of the particular file path, a leaf node in the tree-structured index that includes an entry for the particular file; identifying, in the leaf node, an entry for the particular file path; obtaining, from the entry, the unique file identifier for the particular file in the data store; and using the unique file identifier for the particular file in response to the request.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing space allocation of one or more data storage devices. In some examples, a storage service may identify at least one computer-readable storage device. The computer-readable storage device may, in some examples, have availability level that corresponds to a size of data to be stored. The storage service may also identify a target storage device based at least in part on a capacity of a subset of the storage devices. In some cases, this may be performed for one or more of the devices of the subset. Further, the storage service may store the data in the identified target storage device.

Abstract: A digital asset management system (DAMS) provides an integrated user experience for the rapid and facile manipulation of a users digital assets in an online environment. Spanning the spectrum of digital assets, the DAMS system set forth herein enables users to tag, filter, collect, and show such digital assets in a manner which requires preferably minimal user interaction in order to achieve digital asset management and exhibition. Sharing among friends and family becomes much easier and incoming digital assets may be automatically segregated according to user choices or history to enable easier management of such assets. Recognition of relationships, characteristics, features, and the like between different assets may occur automatically and develop an ongoing fashion to enable association of such similar assets into “collections” or the like.

Abstract: The data storage system according to certain aspects can implement partial file restore, where only a portion of the secondary copy of a file is restored. Such portion may be designated by one or more application offsets for the file. The system may provide an in-chunk index that includes mapping information between the application offsets and the secondary copy offsets. Chunks may refer to logical data units in which secondary copies are stored, and the in-chunk index for a chunk may be stored in secondary storage with the chunk. Because the mapping information may not be provided at a fixed interval, the system can search through application offsets in the in-chunk index to locate the secondary copy offset corresponding to the portion application offset(s). In this manner, the system may restore the designated portion of the secondary copy in a fast and efficient manner by using the in-chunk index.

Abstract: A database recovery and index rebuilding system involves at least one processor coupled to memory, storage, a database, and sets of image copies and post-backup updates. When database reconstruction is required, the processor will i) access the image copies; ii) access post-backup updates to be re-applied; iii) without regard to “ii)”, read data pages to be recovered into objects; iv) concurrently with “iii)”, retrieve and sort log records from the post-backup updates; v) as the data pages to be recovered are read in “iii)”, apply the sorted log records; vi) as “e)” completes and concurrently with “v)”, extract and sort index keys; vii) concurrently with “vi)” and on an individual recovered page basis, write the recovered data pages into the database; viii) when all index keys have been extracted and sorted, initiate database index rebuilds; and ix) when rebuilt, store the rebuilt indexes without regard to whether “vii)” has completed.

Abstract: Methods and apparatuses select service tasks according to allocations of an available usage rate of a common processing resource are described. An ordering relationship is updated among the service tasks. E service task can have a rate of usage of the common processing resource. Each service may be associated with one of multiple task types. An allocation of the available rate of usage may be determined among the task types. The allocation can indicate a portion of the available rate of usage for each task type. The service tasks may be selected according to the allocation and the ordering relationship. At least one of the selected service tasks is associated with each task type associated with the service tasks.

Abstract: In some examples, a method is described. The method may include parting a data item into multiple data units based on update frequencies of the data units. The method may also include identifying a low update frequency data unit from amongst the data units. The method may also include determining a regional sensitive group for the low update frequency data unit based on a historical access pattern of the low update frequency data unit. The regional sensitive group may include a first host. The method may also include replicating the low update frequency data unit. The method may also include pushing the replicated low update frequency data unit to the first host.

Abstract: A method, medium, and system to receive a request to remove a service of an instance of a distributed database system; redistribute data of the service to be removed to at least one remaining service of the distributed database system; save unsaved log segments of the service to be removed in a log backup; register removal of the service to be removed with the distributed database system; delete an entry of the service to be removed from a startup configuration of the distributed database system; halt operation of the service to be removed; and remove the halted service from internal topology information of the distributed database system.

Abstract: A computer-implemented method for reclaiming storage space in deduplicating data systems may include (1) storing at least one non-synthesized backup image within a deduplicating data system by storing at least one data segment of the non-synthesized backup image within a container of deduplicated data segments that is used by the deduplicating data system, (2) storing at least one synthesized backup image within the deduplicating data system by referencing at least one previously stored data segment within the container that is also referenced by a previous backup image within the deduplicating data system, (3) determining that no non-synthesized backup images still reference the container, and (4) removing the data segment of the non-synthesized backup image from the container of deduplicated data segments in response to determining that no non-synthesized backup images still reference the container. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: A log-structured data store implementing data backup may implement variable data replication. Write requests may be received at different storage nodes maintaining respective replicas of a portion of a log for data maintained in the log-structured data store. Log records indicating the write requests may be stored in the respective replicas of the log portions at the different storage nodes. The log records may be sent to a backup data store to be durability persisted as part of an archived version of the log. At some of the storage nodes, in response to determining that the log records have been durably persisted in the backup data store, storage space for the log records may be reclaimed. In other remaining storage nodes, the log records may be retained and made accessible for servicing read requests.

Abstract: A unified context-aware content archive system allows enterprises to manage, enforce, monitor, moderate, and review business records associated with a variety of communication modalities. The system may store an information infoset derived or inferred from one or more documents representing communications according to the variety of communication modalities as interaction transcripts. An interaction transcript represents interactions between participants through the documents rather than the documents themselves allowing for derivation or inference of communication events, chronologies, and mappings to be stored in a common data structure. In one aspect, events correlation is provided between participants of communications that can be established by general time series analysis for the purposes of extracting meaningful statistics and interaction contexts and other characteristics of data. In another aspect, chronological mappings are provided of conversations between an established start and end time frame.

Abstract: In an exemplary embodiment of this disclosure, a method for loading data from a backup image of a database includes selecting a subset statement defining a subset of the data in the database. Tables of the database are identified based on metadata of the database. A target database is written having the structure but not the data of the identified tables. One or more table statements are constructed, by a computer processor, defining a subset of each identified table based on the subset statement. Selected data is unloaded from a backup image into the target database using respective table statements as filters.

Abstract: Updating a central repository with information about malware resident upon a computer system. Upon detecting the malware executing in a virtual machine, a software module, without manual instruction, sends malware manifest data to a central repository over a network. The malware manifest data may comprise a copy of the malware and all versions, including temporary versions, of any files written to, updated by, or accessed by the malware. The central repository may receive, over a network from at least two computer systems, distinct sets of malware manifest data and may subsequently store the sets of malware manifest data.

Abstract: According to certain aspects, a method can include electronically accessing, by a storage manager, a stored synchronization policy specifying at least a first folder associated with a first computing device and a second folder associated with a second computing device; identifying a first file stored in the first folder as a candidate for synchronization; receiving an indication as to a location of the first computing device and the second computing device; in response to a determination that the first computing device is not located within a geographic region, electronically generating and transmitting instructions to the first computing device to remove the first file from the first folder; and in response to a determination that the second computing device is located within the geographic region, electronically generating and transmitting instructions to cause transmission of the first file to the second computing device for storage in the second folder.

Abstract: A processing device performs a search query to produce a search result set having entries having data items. Each data item has an ordinal position. A table, having rows and columns, is displayed in a graphical user interface. Each data item of a particular entry appears in a respective column of the same row of the table. Each column corresponds to the ordinal position of its respective data item. User input is received designating, for each respective column, a field name and an entity definition component type to which the respective column pertains, and stores for each data item of the particular entry an element value of an entity definition. The element has the element name designated for the respective column in which the data item appeared, and is associated with an entity definition component having the type designated for the respective column in which the data item appeared.

Abstract: A method for storing a data object includes identifying data segments of the data object. The method continues with generating key indexes for the data segments. For a data segment, the method continues with accessing data segment key information based on a corresponding key index of the plurality of key indexes to determine whether an encryption key has been generated for a similar data segment. When the encryption key has been generated for the similar data segment, the method continues with using the encryption key to encrypt the data segment to produce an encrypted data segment. The method continues with compressing the encrypted data segment to produce a compressed and encrypted data segment. The method continues with storing the compressed and encrypted data segment in a storage unit of a dispersed storage network (DSN).

Abstract: A system and method of selectively providing encrypted data is provided. Embodiments of the invention may store data in encrypted form on a storage device. Embodiments of the invention may selectively provide encrypted or decrypted data to a requestor of data based on configuration or other parameters. A filter driver or other module or unit may examine a request for, or communication of data from the storage device and may determine if data is to be provided in encrypted or decrypted form. Decrypted data may be provided to a caching system. A filter driver or other module or unit may examine a request for, or communication of data from the caching system. Data provided from the caching system may be selectively encrypted based on configuration or other parameters.

Abstract: A backup agent issues to a virtual machine management service (VMMS) a request to export a virtual machine (VM). The VMMS generates a configuration file associated with the export that cannot be accessed by the backup agent because the file is encrypted. A full backup of the exported VM is performed. A first backup check point corresponding to the full backup is created. After the full backup, a second backup check point is created. The agent issues to the VMMS a request to export changes to the VM between the first and second backup check points. An incremental backup of the exported changes is performed and a backup components document is created. A synthetic full backup based on the incremental and full backups is created and restoration of the VM from the synthetic full backup is based on the information in the backup components document.

Abstract: Identifying virtual machines and errors for snapshots is described. A system receives, from a volume snapshot service, an error message and a corresponding component identifier associated with a backup operation. The system requests a corresponding snapshot status from each component participating in the backup operation. The system receives a corresponding snapshot status from each component participating in the backup operation. The system requests an error identifier from a component corresponding to a snapshot failure status. The system receives the error identifier from the component corresponding to the snapshot failure status. The system identifies a virtual machine based on the component identifier. The system outputs, via a user interface, a report identifying the virtual machine and the error identifier.

Abstract: IT organizations are challenged with maintaining data application and availability while satisfying service level agreements and compliance requirements. Unprecedented data growth is straining IT resources requiring greater investments, while capital and operational expenses are under scrutiny. Given this tension, IT organizations are seeking efficiency and optimization to regain control and meet business SLAs. Example embodiments of the present invention provide a method, an apparatus, and a computer program product for performing automated data protection with information technology (IT) infrastructure visibility. The method comprises receiving input regarding a desired state of a data protection environment for an IT infrastructure and creating a data protection environment according to the desired state of the data protection environment.

Abstract: Described herein are methods, systems and machine-readable media for migrating data between storage devices of a storage array. A metric is used to measure the optimality of candidate data migrations, the metric taking into account capacity balance and proper data striping. Candidate migrations are evaluated against the metric. The candidate migration that ranks as the best migration according to the metric may be carried out. This process of evaluating candidate migrations and carrying out the best candidate migration may be iterated until data is properly distributed among the storage devices of the storage array.

Abstract: A method of specifying behavior among a group of computing tasks included in a request to be performed in a domain of computing resources is disclosed. Method steps include receiving, at a scheduler operably coupled to the domain, a p/f request, the received p/f request including a first group and a first relationship, the first group comprising at least a first p/f group element and a second p/f group element, the first relationship defining a desired behavior of the first and second p/f group elements with respect to each other during performance of the p/f request; determining whether the domain includes available computing resources capable of satisfying the first relationship; and in response to a determination that the domain includes available computing resources capable of satisfying the first relationship, allocating, with the scheduler, at least one available computing resource to fulfill the p/f request.

Abstract: The data storage system according to certain aspects can implement table level database restore. Table level database restore may refer to restoring a database table and its related data without restoring the entire database. The data storage system may use table metadata index to implement table level restore. A table metadata index may be created for each table, e.g., during a backup of the database. The table metadata index for a table can include any type of information for restoring the table and its related data. Some examples of the type of information included in the table metadata index include the following: container for the table, table backup location, system data, table index, table relationships, etc. Table metadata index can make the restoring of tables fast and efficient by packaging information that can be used to restore a table and its related data in an easily accessible manner.

Abstract: A technique for cloud infrastructure backup in a virtualized environment utilizing shared storage includes obtaining a workload input/output (I/O) profile to the shared storage over a time period. An attempt to locate one or more time windows in the workload I/O profile for which a cloud infrastructure backup can be staged is initiated. In response to determining the cloud infrastructure backup can be staged during at least one of the time windows, staging of the cloud infrastructure backup is scheduled during a selected one of the time windows. In response to determining the cloud infrastructure backup cannot be staged during at least one of the time windows, an interference tolerance approach is employed for accessing the shared storage for active workloads and the cloud infrastructure backup during the staging of the cloud infrastructure backup.

Abstract: Techniques to back up data associated with a cluster environment are disclosed. In various embodiments, an indication is received to back up data associated with the cluster. A backup configuration data associated with the cluster is used to back up, in a unified backup operation, one or more save sets associated with virtual resources associated with the cluster and one or more saves sets associated with physical nodes associated with the cluster, including by storing each respective save set in a manner that associates the save set with a virtual or physical node comprising the cluster suite.

Abstract: Techniques for computer system recovery which remotely restore a default partition to a recent state even when an operating system is functioning abnormally. In an example embodiment, a service center computer establishes a first network connection to a monitored computer system. The service center computer configures the monitored computer system to boot from a bootable image file in the monitored computer system and reboots the monitored computer system into an alternate operating system environment of the bootable image file. The service center computer establishes a second network connection to the monitored computer system to restore a recent backup image of the default partition from a diagnostic partition to a default partition. The service center computer establishes a third network connection to the monitored computer system and reboots the monitored computer system to the default partition.

Abstract: Techniques for recovering a virtual machine are disclosed. In one particular embodiment, the techniques may be realized as a method for recovering a virtual machine comprising the steps of selecting a first backup copy of a virtual machine in a first data store, selecting a second data store; enabling use of the virtual machine by reading from the first data store and writing to the second data store, selecting a second backup copy of the virtual machine, and migrating the virtual machine from the second backup copy to the second data store.

Abstract: A technique for cloud infrastructure backup in a virtualized environment utilizing shared storage includes obtaining a workload input/output (I/O) profile to the shared storage over a time period. An attempt to locate one or more time windows in the workload I/O profile for which a cloud infrastructure backup can be staged is initiated. In response to determining the cloud infrastructure backup can be staged during at least one of the time windows, staging of the cloud infrastructure backup is scheduled during a selected one of the time windows. In response to determining the cloud infrastructure backup cannot be staged during at least one of the time windows, an interference tolerance approach is employed for accessing the shared storage for active workloads and the cloud infrastructure backup during the staging of the cloud infrastructure backup.

Abstract: In one embodiment, a method includes establishing at a Fiber Channel over Ethernet (FCoE) Data-Plane Forwarder (FDF), a connection with a controller in communication with a plurality of FDFs, processing at the FDF, control information received from the controller, and forwarding FCoE frames based on the control information. The FDF communicates with the other FDFs in an FCoE fabric and communicates with the controller through a control connection. An apparatus is also disclosed herein.

Abstract: Disclosed is an improved approach for implementing a storage system having an operation log where data writes are initially performed to a fast operation log, rather than to its permanent storage location. A distributed operation log can be employed to replicate the data to multiple locations in the storage system.

Abstract: A method of storing data in a storage system is provided. The method includes writing data, from a virtual machine in a compute node, to a primary storage coupled to the virtual machine and sending a first copy of the data from the compute node to a data node. The method includes writing the first copy of the data from the compute node to an intermediate storage coupled as direct attached storage to the data node, and writing a second copy of data from the intermediate storage to a secondary storage, wherein at least one method operation is performed by a processor. A storage system is also provided.

Abstract: An apparatus, for connection to a storage device, comprising: a module for communication with a software application and operable to receive instructions to copy data to the storage device; a chunking and identifier generation module operable to receive the data, to process the data into one or more chunks, to generate a first chunk identifier, representative of the identity of a first of the or each chunk of data and, upon processing of the data, to initiate the issuance of a confirmation signal to the software application indicating that the data has been copied to the storage device; and an interface for communication with the storage device, wherein the interface is operable to send the first chunk identifier to the storage device, and to send the first chunk of data to the storage device upon receipt of a transfer instruction from the storage device.

Abstract: Disclosed are various embodiments for processing file modifications in a networked storage system. A new version of a file is obtained from a client over a network. The new version of the file is stored in a file system maintained by a networked storage system. If a modification listener is associated with the file, a notification that the file has been modified is sent to the modification listener.

Abstract: Backup management of software environments in a distributed network environment includes: Creating a platform specific software environment according to a predefined software environment definition by a user in the network environment; automatically creating a generalized backup of the software environment by extracting platform independent data contents and platform specific transformation rules; automatically calculating enhanced meta data describing the generalized backup; assigning the enhanced metadata with the predefined software environment definition; registering the generalized backup with the enhanced metadata in a backup storage accessible by a plurality of distributed users connected via said network environment, creating a software environment according to a predefined software environment definition by automatically checking the backup storage for an available generalized backup by means of the predefined software environment definition.

Abstract: Systems and user interfaces enable integration of data items from disparate sources to generate optimized packages of data items. For example, the systems described herein can obtain data items from various sources, score the data items, and present, via an interactive user interface, options for packaging the data items based on the scores. The systems may include artificial intelligence algorithms for selecting optimal combinations of data items for packaging. Further, the interactive user interfaces may enable a user to efficiently add data items to, and remove data items from, the data packages. The system may interactively re-calculate and update scores associated with the package of data items as the user interacts with the data package via the user interface. The systems and user interfaces may thus, according to various embodiments, enable the user to optimize the packages of data items based on multiple factors quickly and efficiently.

Abstract: Provided are a computer program product, system, and method for building a metadata index from source metadata records when creating a target volume for subsequent metadata access from the target volume. A copy operation is initiated to copy the source volume to the target volume, wherein the target volume is created with a target volume name different from a source volume name of the source volume. Source volume metadata having records including information on data sets in the source volume is copied to target volume metadata. A metadata index is generated indicating locations of the records in the target volume metadata having information on the data sets in the target volume. The metadata index is used to access the records in the target volume metadata when accessing data sets in the target volume.

Abstract: Embodiments of the invention are directed to systems and methods for maintaining coherency between different entities in a distributed system. A coherency module automatically detects a change in state in a first entity, wherein the change in state relates to a change in functional code in the first entity. A synchronization message is transmitted to a second entity to synchronize data in the second entity with data in the first entity as a result of the change in state. The second entity is configured to synchronize the data in the second entity with the data in the first entity after receiving the synchronization message.

Abstract: A method of providing access control to a database accessible from a user interface is implemented at a policy enforcement point, which is located between the database and the user interface and includes the steps of: (i) intercepting a database query; (ii) assigning attribute values on the basis of a target table or target column in the query, a construct type in the query, or the user or environment; (iii) partially evaluating an access-control policy defined in terms of said attributes, by constructing a partial policy decision request containing the attribute values assigned in step ii) and evaluating the access-control policy for this, whereby a simplified policy is obtained; (iv) deriving an access condition, for which the simplified policy permits access; and (v) amending the database query by imposing said access condition and transmitting the amended query to the database.

Abstract: A computer-implemented method for accelerating backup operations may include (1) identifying a volume of data that may be stored by a computing system attached to a network and that may be designated for a backup to a deduplicated data store, (2) monitoring traffic on the network for at least one data object destined for the volume of data, (3) copying the data object from the traffic on the network and (4) seeding the deduplicated data store with the data object before the backup to the deduplicated data store. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Techniques are disclosed for consuming and integrating with representational state transfer (REST) based services. In one embodiment, a multiple-step extract, transform, load (ETL) job includes one or more REST steps. Each step in the ETL job performs a particular well-defined task, and the REST step in particular constructs REST requests, invokes REST services, and retrieves REST responses. The REST step may be mapped to a vector of inputs such that multiple REST service calls are made for each of the inputs. Further, a reusable connection may be employed so that one connection can be used to make multiple REST service calls. URLs used to make REST service calls may be dynamically constructed based on input data parameters and ETL job parameters. Additionally, a graphical user interface (GUI) allows users to configure aspects of a REST step.

Abstract: A method, apparatus and computer program product are provided for preparing and installing update packages for compressed files. In the context of a method, a method for preparing an update package is provided that includes receiving an original file and a modified file, causing the original file and the modified file to be decompressed, and generating one or more delta files based on the decompressed original file and the decompressed modified file. A corresponding method for installing an update package is also provided that includes receiving the update package comprising one or more delta files corresponding to an original file, causing the original file to be decompressed, generating one or more modified subfiles based on the one or more delta files and the decompressed original file, and generating a compressed modified file by compressing the one or more modified subfiles.

Abstract: A control apparatus that causes data in a first storage unit to be written in a second storage unit, with power supplied from a sub power supply, when supply of power from a main power supply is discontinued, the control apparatus includes a remaining feed duration obtaining unit that obtains remaining feed duration during which the sub power supply can supply the power; and a retry count setting unit that sets a maximum retry count for writing the data from the first storage unit to the second storage unit, based on the remaining feed duration obtained by the remaining feed duration obtaining unit, when an error occurs while the data is being written.

Abstract: Various embodiments for differentiating between data and stubs pointing to a parent copy of deduplicated data. Undeduplicated data is stored with a checksum of an initial value. A stub pointing to the parent copy of the deduplicated data is stored with an additional checksum of a differing, additional initial value.

Abstract: A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Abstract: Techniques are presented for versioning files. Base files are acquired for files on a primary volume. At configurable points in time changes to the files are noted on different volumes, these changes are represented as deltas. The deltas are stored on an archive volume and represent versions and a version history for the files.

Abstract: A source volume is restored to the time of a selected incremental backup volume without writing over all of the data blocks in the source volume. If the source volume has valid data blocks and there are valid records of the changes to these valid data blocks and the valid records are consistent with the state of the source volume and indicate that certain valid data blocks did not change since the selected incremental backup, then the restoration does not write over the valid data blocks that did not change since the selected incremental backup. Otherwise, data blocks are recovered from the selected incremental backup and one or more backups prior to the selected incremental backup. For example, data blocks are recovered by scanning incremental backups in reverse chronological order to find the most recent version of the block in these backups.

Abstract: A database recovery method involves: a) accessing image copies; b) accessing updates to be re-applied to reconstruct the database; c) reading into objects, from the image copies, data pages to be recovered; d) concurrently with “c)” retrieving and sorting log records from the updates; e) as the data pages are read into the objects in “c)”, applying the sorted log records, in timestamp order; f) as “e)” completes for individual pages, and concurrently with “e)” with respect to pages for which applying is not complete, extracting and sorting index keys, from pages for which the applying is complete, until all index keys have been extracted and sorted; g) concurrently with “f)”, writing recovered pages into the database; h) initiating a rebuild of database indexes using the extracted and sorted index keys; and i) when the database indexes have been rebuilt, writing them to storage without regard to whether “g)” completed.

Abstract: A storage system is described. The storage system includes an object store having a plurality of objects to implement a quota policy or a part of a relational database. The plurality of objects include a first plurality of immutable objects, a second plurality of immutable objects and a head object. The first plurality of immutable objects contain information kept within the database. The second plurality of immutable objects contain a hierarchy of pages for accessing the first plurality of immutable objects. The plurality of objects also include a mutable object that contains a mapping table that maps identifiers of the pages to identifiers of objects of the second plurality of immutable objects. The information is one of: quota policy information; information within a table of a relational database; a stripe of a thin provisioned file.

Abstract: In one aspect, a method includes generating a secure connection with a replication system using credentials, receiving a user-initiated command to control the replication system, generating a connection object, determining if the connection to the replication system has been initiated, sending the command to the replication system if the connection is initiated, obtaining input from a character stream to process and processing data for use by a backup system.

Abstract: The system and method described herein may provide a virtualized replication and high availability environment. In particular, a virtualized production server may run one or more virtual machines in one or more child partitions and have a replication and high availability engine installed in a parent partition. The replication and high availability engine may automatically discover the virtual machines running in the child partitions and automatically synchronize all files associated with the virtual machines to a virtualized replica server.