Abstract: Methods, computer systems, and computer program products for replicating data are provided. Meta-data and data associated with the meta-data are received at a first data server. A new portion of the data that is not already stored on the first data server is identified. The new portion of the data is stored on a second data server before the meta-data is stored on the second data server.

Abstract: A method, a system, an apparatus, and a computer readable medium for transmission of data across a network are disclosed.

Abstract: Described herein are methods and systems for providing software development services more efficiently. Re-computation of results each time a service request is received can be avoided by maintaining a cache of results from having processed requests. Results are stored under a unique mapping of request-results pairs that at least in part rely on a file fingerprint hash of the contents of the input files related to the request. In network environment with a client requester and service provider unnecessary transmission is also avoided by first presenting results for requests are in form of a unique file identifier identifying one or more files holding the content of the results. The file identifier is used to search a cache local to the requesting client prior to requesting transmission of the results. The file identifier may also include an indicator for indicating a location from which the results may be retrieved.

Abstract: A computer-implemented method for data-selection-specific data deduplication associated with a single-instance-storage computing subsystem may comprise: 1) detecting a request to store a data selection to the single-instance-storage computing subsystem, 2) identifying a data-selection-specific fingerprint set associated with the data selection and stored on a storage device, and 3) utilizing the data-selection-specific fingerprint set associated with the data selection for data deduplication associated with the request to store the data selection to the single-instance-storage computing subsystem. Other exemplary data deduplication methods, as well as corresponding exemplary systems and computer-readable media, are also disclosed.

Abstract: A system stores file system metadata in a manner that scales to a large number of entries per “capture,” where a capture is metadata of all of the files and their associated directories contained in a logical file system at a given time. The file system metadata is stored, in tangible computer readable media, in data structures that are optimized to the unique characteristics of file system metadata and, particularly, how it is desired to access and process file system metadata. Thus, the technical effect is that processing storing and processing of the file system metadata is highly optimized, such that time, storage and processing power to process the file system metadata is drastically reduced.

Abstract: Embodiments of the invention determine if a user preference or other piece of data is being synchronized too frequently. If the user preference is being synchronized too frequently, synchronization of the user preference is throttled to prevent it from synchronizing for some number of synchronization cycles. If the user preference rarely changes, the user preference is rewarded by throttling it less often.

Abstract: An application server in a multi-tenant computing system is enabled access to a first set of tables and views in a database according to a first schema. The views in the first schema correspond to a table in a second schema. Thereafter, the table in the second schema is modified to add a column. Views of the modified table as defined by the second schema are generated in a third schema so that the third schema defines a second set of tables and views in the database. Subsequently, access by the application server to the second set of tables and views in the database is enabled and access by the application server to the first set of tables and views in the database is prevented. Related apparatus, systems, techniques and articles are also described.

Abstract: A document is stored at each of a number of database replicas communicatively connected with a newly created database replica. The newly created database replica stores an indicator for the document denoting the database replicas at which the document is stored. A probable time to retrieve the document from each database replica is retrievable from the newly created database replica. For each database replica, a real-time analysis of network parameters in relation to the database replica is able to be performed based on the probable time retrieved. This analysis yields an updated probable time to retrieve the document from the database replica. This enables the document to be retrieved from the database replica having the lowest updated probable time.

Abstract: In each of a number of passes to deduplicate a data object, a transaction is started. Where an offset into the object has previously been set, the offset is retrieved; otherwise, the offset is set to reference a beginning of the object. A portion of the object beginning at the offset is deduplicated until an end-of-transaction criterion has been satisfied. The transaction is ended to commit deduplication; where the object has not yet been completely deduplicated, the offset is moved just past where deduplication has already occurred. The object is locked during each pass; other processes cannot access the object during each pass, but can access the object between passes. Each pass is relatively short, so the length of time in which the object is inaccessible is relatively short. By comparison, deduplicating an object within a single pass prevents other processes from accessing the object for a longer time.

Abstract: Various techniques for improving the performance of restoring deduplicated data files from a server to a client within a storage management system are disclosed. In one embodiment, a chunk index is maintained on the client that tracks the chunks remaining on the client for each data file that is stored to and restored from the storage server. When a specific file is selected for restore from the storage server to the client, the client determines if any local copies of this specific file's chunks are stored in files already existing on the client data store. The file is then reconstructed from a combination of these local copies of the file chunks and chunks retrieved from the storage server. Therefore, only chunks that are not stored or are inaccessible to the client are retrieved from the server, reducing server-side processing requirements and the bandwidth required for data restore operations.

Abstract: A method and system for identifying changes to a data set, such as data within a mailbox, and performing actions based on the identified changes is discussed. In some examples, the system receives an indication of a change to a mailbox, creates a change journal entry for the change, and identifies data to be copied via the change journal entry. In some examples, the system leverages the change journal to associate messages with changes to a mailbox.

Abstract: A combination of synchronous and asynchronous replication of data is used to replicate a local database to a replicated database. The typical tradeoff between synchronous and asynchronous replication is optimized by using hybrid replication, which is to use synchronous replication for inserting new data and asynchronous replication for updating existing data. The combined use of synchronous and asynchronous in this manner provides an efficient replicated database where the replicated database can tolerate some delay in data updates but requires no data loss of new data.

Abstract: Data is recursively migrated during uptime from a first table to a second table in a first schema in a database while taking into account changes to the first table in a change table. The database has first and second application servers respectively connecting to the database using first and second schemas and running first and second versions of a software program. Downtime can be initiated to migrate the remaining entries in the change table to the second table, delete the first table, and rename the second table to have a same name as the first table prior to deletion so that during uptime the second application server connects to the second table and runs the second version of the software program. Related apparatus, systems, techniques and articles are also described.

Abstract: A system, method, and computer program product for efficient synchronization of data structures. A method includes maintaining a source data structure and a corresponding accountability table in the source data processing system, where at least a portion of the data structure is to be synchronized with a target data structure in a target data processing system. The method includes receiving a synchronization command, and in response to receiving the synchronization command, identifying a first plurality of synchronization objects in the source data structure, using the accountability table, that also exist in the target data structure and have been modified in the target data structure. The method includes identifying a second plurality of synchronization objects in the source data structure that have been newly-created or deleted and that either reference or are referenced by an object in the accountability table.

Abstract: Embodiments are directed to providing database access during database reconfiguration and to maintaining replication connections during database reconfiguration. In an embodiment, a computer system establishes multiple quorum sets of replicas to replicate the data of a data partition. The quorum sets of replicas ensure that at least a minimum number of replicas are operating to commit pending transactions during partition reconfiguration. The computer system determines that a data partition reconfiguration has been initiated and provides access to the data partition's data during reconfiguration of the data partition using at least a quorum of replicas in each of the quorum sets of replicas.

Abstract: A method, and apparatus for software and resource management with a model-based architecture.

Abstract: A computer-implemented method for relocating deduplicated data within a multi-device storage system. The method may include identifying a set of deduplicated data units stored on a first device of the multi-device storage system. Each data unit in the set of data units is referred to by one or more deduplication references. The method may also include procuring reference data that indicates, for each data unit in the set of deduplicated data units, the number of deduplication references that point to the data unit. The method may further include using the reference data to select one or more data units from the set of deduplicated data units for relocation to a second device in the multi-device storage system and relocating the one or more data units to the second device in the multi-device storage system. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Software is provisioned by providing a repository that contains a software trove. A local changeset is generated that represents differences between a version of the software trove on the repository and a corresponding version of the software trove on a local system.

Abstract: A replication process using snapshots to maintain information regarding changes made to a production data set following a point in time. In some embodiments, the production data set may be a file system and the snapshot may store information relating to changes made to both previously-allocated and newly-allocated data blocks. When a change is made to a previously-allocated data block, the original contents for the data block may be copied to a save volume prior to being overwritten by the new data, while for a newly-allocated data block no data may be copied to the save volume. An entry may also be created in a map identifying each changed data block. A replication process may then read the block map to identify data blocks storing new data to should be replicated to a replication data set, such that the replication data set remains a copy of the production data set.

Abstract: A document stored at each of a number of database replicas communicatively connected with a current database replica is desired to be opened. The current database replica is initially opened. The current database replica stores an indicator for the document denoting the database replicas at which the document is stored. A probable time to retrieve the document from each database replica is retrieved from the current database replica. For each database replica, a real-time analysis of network parameters in relation to the database replica is performed based on the probable time retrieved. This analysis yields an updated probable time to retrieve the document from the database replica. The document is thus retrieved from the database replica having the lowest updated probable time.

Abstract: Graph-based modeling apparatus and techniques are disclosed. Based on a model including model nodes that represent components of a modeled system, operational dependencies between model nodes, and model edges that interconnect the nodes and represent relationships between the components in the modeled system, subset computations are performed to compute subsets of the model nodes that can impact operational dependencies between other model nodes. When the model changes, a determination is made as to whether an incremental subset computation should be performed for one or more particular operational dependencies between model nodes in the changed model, and if so, an incremental subset computation is performed. Otherwise, a full subset computation or no subset computation might be performed. In this manner, model changes are considered on a case-by-case basis to determine an extent, if any, to which subsets should be re-computed.

Abstract: Various methods and systems for performing data deduplication when encrypting data for storage on an untrusted storage system are disclosed. One method involves generating an encryption key for use in encrypting data and generating an identifier for the data. Generation of the encryption key is based upon a hash of the data to be encrypted. Similarly, generation of the identifier is based upon the hash of the data to be generated. For example, the identifier can be generated by hashing the encryption key. The method then involves detecting whether an encrypted copy of the data is already stored by a storage system, based upon the identifier, and selectively encrypting the data, based upon whether the encrypted copy of the data is already stored by the storage system.

Abstract: A mechanism for efficient delayed replication is described. A method of embodiments of the invention includes detecting input/output (I/O) requests issued by a software program running on a computer system, and sorting the I/O requests into I/O batches according to flight overlapping of the I/O requests. An I/O batch includes a set of flight-overlapped I/O requests of the I/O requests. The set of flight overlapped I/O requests includes two or more I/O requests that are issued, at least partially, in parallel. The method further includes replicating the I/O requests to a storage medium coupled to the computer system.

Abstract: Methods and apparatus, including computer program products, for distributing data in a data management system. The technique includes identifying one or more objects in a central data store for distribution, determining if a routing exists for at least one object of the one or more objects, and distributing the at least one object to one or more target systems specified by the routing. The one or more objects include master data objects for use by all systems in a data management system, and the one or more target systems are part of the data management system.

Abstract: A hierarchical database stores data for the database in a plurality of areas. A disclosed technique allows reorganization of one or more areas of the database without stopping the entire database. The areas to be reorganized are first stopped, then the areas are unloaded, reorganized, and reloaded, before restarting the reorganized areas. In-memory control blocks for the areas are updated to indicate to the database software that the areas have been reorganized, without stopping the entire database.

Abstract: In-band commands may be associated with a particular consistency interval and may indicate requested actions to be performed for that consistency interval. An application may desire to perform actions, such as additional backup, snapshots, etc. on stored data, when that data is in a consistent state from the application's point of view. In order to ensure that the data is in a consistent state, a consistency interval may be created on demand. A node may request a consistency interval by sending a consistency request message to a consistency interval coordinator, which in turn, establishes the consistency interval with all nodes in the distributed environment. After sending all write requests for the consistency interval, the node may then send the command message. Command messages may be stored in consistency logs along with write requests and a replication target, or other device, may read both the write requests and the command message.

Abstract: A system configured to optimize access to stored chunks of data is provided. The system comprises a vLUN layer, a mapped LUN layer, and a mapping layer disposed between the vLUN and the mapped LUN. The vLUN provides a plurality of logical chunk addresses (LCAs) and the mapped LUN provides a plurality of physical chunk addresses (PCAs), where each LCA or PCA stores a respective chunk of data. The mapping layer defines a layout of the mapped LUN that facilitates efficient read and write access to the mapped LUN.

Abstract: Document identification tags are assigned to documents to be added to a collection of documents. Based on query-independent information about a new document, a document identification tag is assigned to the new document. The document identification tag so assigned is used in the indexing of the new document. When a list of document identification tags are produced by an index in response to a query, the list is approximately ordered with respect to a measure of query-independent relevance. In some embodiments, the measure of query-independent relevance is related to the connectivity matrix of the World Wide Web. In other embodiments, the measure is related to the recency of crawling. In still other embodiments, the measure is a mixture of these two. The provided systems and methods allow for real-time indexing of documents as they are crawled from a collection of documents.

Abstract: A peer-to-peer file sharing client with background file sharing is provided in a segmented peer-to-peer file sharing network. Each file sharing participant may designate an amount of bandwidth and/or storage space for background file sharing. Peer-to-peer file sharing clients then share file data and content in the background automatically. The client may participate in additional swarms, in the background, to generally increase the number of peers in file sharing networks, thus increasing the speed of downloading desired files for other users.

Abstract: According to the present invention, techniques for controlling copying of logical volumes within a computer storage system are provided. A representative embodiment includes a plurality of storage devices controlled by a control unit, one or more processors, and a buffer memory for temporarily storing data read from the storage devices within the control unit. The storage devices can be addressed as logical volumes.

Abstract: Systems and methods that supply a replication layer/agent that is generic to supporting a plurality of storage configuration as part of a distributed store. Such distributed store employs a Common Availability Substrate (CAS) for data transport and consistency, to render the distributed store scalable and available. Such an arrangement enables continuous operation of the store, while the replication subsystem creates new replicas (e.g., for load balancing, failover, and the like).

Abstract: Collection-based object replication for a system that includes a client computing device (client) connected to a server and multiple data storage nodes. In certain cases, a data storage node generates a replica of multiple replicas of a collection. The collection is a unit of data placement, access, replication, and repair. Other data storage nodes are also configured with a respective replica of the multiple replicas. The data storage node verifies whether an object received directly from the client for storage in the collection has been fully replicated by the other data storage nodes in respective replicas.

Abstract: A method for allowing client computers in a distributed object-based data storage system to interpret higher revision data structures. In one embodiment, clients request layout maps of objects to be accessed from respective storage managers in the system and then construct and execute directed acyclic graphs (DAG) to accomplish desired I/O operations for data access. When a client computer running an older software revision is not able to interpret a particular data organization supplied by the storage manager that was designed after that client's software was installed, the client sends to the storage manager a description of the desired data access operation. The storage manager, in turn, interprets a data layout scheme for a client on a per-access basis and returns to that client a DAG containing a data structure that describes the sequence of I/O operations necessary to effect the data access. The client can then execute the storage manager-supplied DAG to carry out the data access.

Abstract: A computer is programmed to parse a network that interconnects databases in a distributed database system, into a number of paths, wherein each path starts in a source database that is a source of changes and ends in a destination database that uses the changes to replicate the source database. The computer identifies databases that occur within each path in the network, and queries the identified databases to obtain statistics for each path. Statistics along each path are analyzed automatically, to identify a component in each path as a candidate for further investigation. The identified component may be marked as a bottleneck e.g. if there are no other paths connected to it. Alternatively the identified component may be a source of changes for another path that has a bottleneck or that has another candidate for further investigation. The identified bottleneck is displayed to the user with recommendations for improvement.

Abstract: Embodiments of the invention may be used to increase query processing parallelism of an in-memory database stored on a parallel computing system. A group of compute nodes each store a portion of data as part of the in-memory database. Further, a pool of compute nodes may be reserved to create copies of data from the compute nodes of the in-memory database as part of query processing. When a query is received for execution, the query may be evaluated to determine whether portions of in-memory should be duplicated to allow multiple elements of the query (e.g., multiple query predicates) to be evaluated in parallel.

Abstract: The disclosure discloses a method and system for reducing an amount of data transfer in data synchronization, in which the method comprises: identifying different data values in each data column within a changed source data table; coding data values in each data column, wherein, for each data column, different data values have different coding values; identifying different ChangeType and ChangeBits in data change history information; and generating a change matrix by using the coding values of different data values in each data column according to different ChangeType and ChangeBits. With the change matrix, data having the same change for different rows within an initial data change are merged into a same change row within the change matrix, thereby reducing amount of data transfer and improving synchronization efficiency.

Abstract: In some embodiments, it is determined that a primary node has been elected from among a plurality of nodes in a replica set of nodes. The primary node accepts and processes client data access requests. The replica set includes the primary node and other secondary nodes. The primary node receives client data access requests. The primary node assigns a data sequence number to each client data access request that mutates state in the order the client data access requests are received. The data sequence numbers include a linearized processing order that is to be followed by each of the nodes in the replica set. The primary node sends the mutating client data access requests including any corresponding data sequence numbers to the secondary nodes. The primary node receives, from a threshold number of secondary nodes, an acknowledgement indicating reception of the client data access request. The primary node commits the data mutating access request.

Abstract: Systems for performing inline wire speed data deduplication are described herein. Some embodiments include a device for inline data deduplication that includes one or more input ports for receiving an input data stream containing duplicates, one or more output ports for providing a data deduplicated output data stream, and an inline data deduplication engine coupled to said one or more input ports and said one or more output ports to process input data containing duplicates into output data which is data deduplicated, said inline data deduplication engine having an inline data deduplication bandwidth of at least 4 Gigabytes per second.

Abstract: A method and apparatus for obtaining data of a cache node in a tree-structured cluster is described. In one embodiment, a query for data in the cache node of the tree-structured cluster is received. A determination of whether the data is stored in the queried cache node is made. An inquiry of other cache nodes in the cluster for the data is performed. An instance of the data from a cache node storing the data is replicated to the cache node receiving the query.

Abstract: A method and apparatus for efficiently logging changes to a file system are disclosed. A file system filter driver intercepts I/O requests directed to a file system, and analyzes the I/O requests to determine whether the requests are write or modify requests directed to a file included in a list of files to be monitored for changes. If the I/O request is directed to such a file, the file system filter driver logs the I/O request to a b-tree structured change log file. Furthermore, in one embodiment, the file system filter driver is configured to log file system changes to multiple log files to coincide with snapshot operations, such that the file system changes to be included in each snapshot are included in separate log files.

Abstract: An improved system and method for loading records into a partitioned database table is provided. A translation of records may be generated from a set of source partitions to a set of target partitions by generating a bipartite graph, determining a maximal matching using dynamic programming for a chain of nodes remaining in the bipartite graph after removing singleton edges, and generating a maximal matching after adding back the singleton edges for translation of records from the set of source partitions to the set of target partitions. The partition translation may be executed by traversing from top to bottom the set of source partitions and the set of target partitions in record key order to generate an optimal sequence of operations to transfer the records from the set of source partitions to the set of target partitions.

Abstract: Methods, systems and apparatus, including computer program products, for propagating data across shards of a database system are described. According to one method, a first request that includes instructions to place a second request in a queue of a second shard is placed in a queue of a first database shard, and the change is committed to the records of the first shard. A queue processor polling the queue fetches and dispatches the first request, placing the second request in a queue of a second database shard. The second request includes instructions for making the change to the second shard. A queue processor of the second database shard polls the queue of the second database, fetches the second request, and dispatches it to make the change on the second database shard.

Abstract: Embodiments of the present invention address deficiencies of the art in respect to structured content storage and provide a novel and non-obvious method, system and computer program product for managing structured content stored in a BLOB. In an embodiment of the invention, a performance optimized structured content management system can include a content repository, a content manager configured to provide access to structured content in the content repository and multiple different performance optimized containers disposed in the content repository. Each of the containers can store a portion of the structured content, and each of the containers can include a flattened form of original structured content in a primary binary large object (BLOB) and a parsed form of the original structured content in a secondary BLOB, the parsed form of the original structured content in the secondary BLOB indexing the flattened form of the original structured content in the primary BLOB.

Abstract: Architecture that provides the capability to automatically (e.g., dynamically) reorganize (repartition) an existing partition by dividing (splitting) or recombining (merging) logical databases. This reorganization can be performed to logical databases belonging to the same customer, and based on the partitioning of the tables in these databases. This can include not only splitting secondary replicas of a partition or merging secondary replicas of the partition, but also splitting off secondary replicas of the partition to create a new partition and merging two partitions into one partition. Moreover, these operations can occur while the logical databases are accepting workload (online).

Abstract: Systems and methods for partitioning information across multiple storage devices in a web server environment. The system comprises a web server database which includes information related creating a web site. The information is divided into partitions within the database. One of the partitions includes user information and another of the partitions includes content for the web site. Portions of the content for the web site is replicated and maintained within the partition including the user information. Further, a portion of the user information is replicated and maintained in the partition where the content for the web site is maintained. The methods include dividing information into partitions, de-normalizing the received data and replicating the data portions into the various web site locations.

Abstract: Replica placement in a network of nodes is provided. Nodes are selected for replica placement to satisfy location-based preferences. Additionally, nodes are selected for replica placement to spread replicas of the same data over different fault domains and upgrade domains. In some instances, nodes may be selected for replica placement based on load-based information for the nodes.

Abstract: An information distribution system is provided with a plurality of information processing devices that are capable of communication with each other via a network, and that store shared information so that the information can be shared by the plurality of information processing devices. The shared information is shared information that has a public release time that is before the current time.

Abstract: A system and method for legacy system component incremental migration from a legacy system to a new system comprises a read-only ghost or shadow in the new system. When changes are made in the legacy system, the changes are incrementally and automatically migrated to the new system allowing the legacy system and the new system to maintain availability during the migration. The concept of “mastership” is used where a component exists in the “New System”, but is actually mastered in the “Legacy System”. By allowing a sub-component to exist as a read-only ghost or shadow in the new system, and still be mastered/edited in the legacy system, a one-way bridge is provided so that the two systems can interact. The synchronization of the legacy system component is managed so that the ghost component is automatically updated when changes are made to the legacy system component and there is little maintenance that the user needs to do to create the bridge between the two systems.

Abstract: Methods and systems are provided for intelligently managing project data in a project management application. According to various aspects of the invention, project management data may be loaded and saved to a local cache of a client computer, such as a hard disk cache. The invention also provides a mechanism for reducing the size of data downloaded from and sent to a project server during load and save times.

Abstract: One embodiment is a method that builds a target database with transaction logs from an online source database. The transaction logs include both existing data in the source database and updates that occur to the source database while the target database is being built.