Abstract: Various techniques for temporal optimization of data operations using distributed search and server management are described, including configuring a host group using a manifest file, receiving a data file at a platform using an application programming interface, parsing the data file to generate a processed data file, the processed data file having a time characteristic associated with the data file, storing the data file at a location by comparing the time characteristic to the time range to determine the class, the server type, and the server on which to store the data file, reconciling the host group to determine whether the data file has changed, and modifying the data file by performing a data operation on the data file if the reconciling indicates the data file has changed.

Abstract: A method for learning queries in automated incident remediation is performed by one or more computing devices, each comprising one or more processors. The method includes parsing at least a portion of incidents from an incident log based at least in part on one or more incident types associated with each incident from the portion of the incidents, identifying parameters associated with a plurality of queries, grouping the plurality of queries into a plurality of query groups based at least in part on the identified parameters, identifying a new incident added to the incident log, and generating an automated query based at least in part on a similarity between the new incident and a prior incident.

Abstract: Log record analysis based on reverse engineering of log record formats is disclosed. A computing device receives log record template data associated with a log record template comprising a plurality of field identifiers for a log record. The log record template data is analyzed to identify the plurality of field identifiers. Based on the log record template data and the plurality of field identifiers, a log record analysis function is automatically generated that is configured to receive a log record that was generated based on the log record template, extract a plurality of log data items from the log record, each log data item corresponding to one of the plurality of field identifiers, and output information that identifies the plurality of field identifiers and the plurality of log data items.

Abstract: A software framework is disclosed. The framework defines a system of a plurality of independent processors including compatible processors. A composite processor is created from the compatible processors.

Abstract: A checkpointing mechanism by which in-memory data structures are copied from computation nodes (200) to staging nodes (700) by using RDMA, checkpoints are made and kept in memory in the staging node (700), and then asynchronously copied to non-volatile storage (150). In contrast to previous approaches, checkpoints remain in volatile memory (740) as part of the checkpointing mechanism. As a result, recovery from checkpoint is potentially faster, since the required checkpoint may be already in memory (740) in the staging node (700). An automatic and customisable mechanism is provided to control when the checkpointing process is triggered. As an alternative to copying an object through the network, the object in memory can be updated to a newer version of the object by applying the chain of changes made in the object in the corresponding computation node (200).

Abstract: An executable utility is injected into cloud-based virtual machines (VMs) that are subject to backups by a data storage management system tasked with protecting the cloud-based VMs and their associated data. The utility is injected into a target VM which is “live” and operating. The utility analyzes the VM's live volume to discover data extents therein, and for each extent computes a respective checksum and determines whether the extent is a “hole.” Afterwards, checksums help identify changed data in successive snapshots of the live volume, so that only changed data will be read and backed up in incremental backups. Time is saved in performing the backup operation first by pre-warming the backup's source volume in parallel with the utility analyzing the live volume, and second by skipping read operations for extents unchanged since a preceding backup. The resulting incremental backup operation is sped up as compared to prior art approaches.

Abstract: Database performance may be improved by creating additional backups from an existing backup, rather than the original database or audit log source. A second backup may be created from a first backup by copying the files from the first backup to the second backup. The replication from a first backup to a second backup with modification of the file history information allows identical backups to exist on different types of storage devices, including virtual storage devices. Continue functionality may be implemented during a move function to restart replication at a new starting point while retaining the existing move set. Tape-type-override functionality may be implemented during the replication to replace the tape type for the backups with another compatible type while retaining the original type for history file entries, tape trailers, and messages.

Abstract: Techniques for providing fault-tolerant stream processing. An exemplary technique includes writing primary output events to a primary target and secondary output events to one or more secondary targets, where the primary output events are written by a primary server and the secondary output events are written by one or more secondary servers. The technique further includes receiving an election of a new primary server from a synchronization system upon a failure of the primary server, where the new primary server is elected from the one or more secondary servers. The technique further includes determining, by the new primary server, the primary output events that failed to be written to the primary target because of the failure of the primary server, and writing, by the new primary server, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

Abstract: Managing application programming interface (API) execution logging is provided. It is determined whether a log trigger definition corresponding to a log include definition is satisfied. In response to determining that the log trigger definition corresponding to the log include definition is satisfied, an execution log of the API is transferred from memory to a file in a storage device. The execution log of the API and a set of API definitions in the file is analyzed using natural language processing to identify a usability context pattern corresponding to the API. A specific set of API log definitions for the API is recommended based on the identified usability context pattern corresponding to the API.

Abstract: Recovery of a database system by taking the database system offline is initiated. Thereafter, recovery operations specified by a redo log of the database system are replayed. A cleanup log is generated that identifies cleanup operations occurring during the replay of the recovery operations for garbage collection. Concurrent with the startup of the database, garbage collection of the cleanup operations as specified in the database savepoint is initiated. In addition, concurrent with the replay of the recovery operations, garbage collection of the cleanup operations specified by the cleanup log is initiated. The database system is later brought online after all of the recovery operations are replayed.

Abstract: According to certain aspects, systems and methods are provided for automating recovery of a networked data storage environment. For instance, a system can generate a data recovery package configured to automatically carry out a process for recovering a data storage environment and/or associated data. The content of the particular workflow depends on the data storage environment, and can be defined by a user.

Abstract: Methods and apparatus are provided to categorize and process an incomplete transaction. A transaction pattern table is provided that includes a plurality of records, each record being defined by a transaction pattern identifier. Each record includes a transaction pattern and a metric of occurrence of the transaction pattern. An incomplete transaction is automatically compared to the transaction patterns in the table and a subset of transaction patterns in the table are identified that are potential candidates for matching the incomplete transaction. The incomplete transaction is categorized using the metric of occurrence of the transaction pattern for each transaction pattern in the subset. The incomplete transaction is then processed using the categorization of the incomplete transaction.

Abstract: Updates to a hierarchical data structure may be selectively replicated to other replicas of the hierarchical data structure. An update for a hierarchical data structure may be received and committed to the hierarchical data structure. A determination as to whether any other replicas of the hierarchical data structure have permission to receive the update may be made. For those replicas of the hierarchical data structure with permission to receive the update, the update may be provided to the replicas and committed to the replicas. Different types of replication techniques may be implemented, such as pull-based replication techniques or push-based replication techniques. Replication permissions for objects of the hierarchical data structure may be individually defined, in some embodiments.

Abstract: Methods, systems, and computer-readable media for transactionally consistent backup of partitioned storage are disclosed. Backup operations are initiated for a first and second partition of one or more data stores. The position of a first marker is determined in a first log associated with the first partition. The first log indicates a sequence of changes for the first partition. The position of a second marker is determined in a second log associated with the second partition. The second log indicates a sequence of changes for the second partition. The first or second marker is determined to violate a transactional boundary across the first and second partitions. A position of at least one of the markers is modified such that the markers do not violate the transactional boundary.

Abstract: A computer-implemented method includes, by a streaming analytics component, identifying a source database table and a target database table. The target database table includes one or more target database rows and one or more target database columns. The method further includes identifying a change stream including a plurality of row-level modifications that cause the target database table to replicate the source database table. The method further includes determining that each row-level modification affects one or more common columns of the target database columns, wherein the common columns exhibit one or more common values for those of the target database rows that are affected by the row-level modifications. The method further includes generating, based on the common values and the common columns, a reconstructed modification statement and sending the reconstructed modification statement to an apply component. A corresponding computer program product and computer system are also disclosed.

Abstract: A journal optimizer in a computer database system with an adaptive journal mechanism. The adaptive journal mechanism dynamically adjusts adaptive parameters of the journal optimizer to optimize the journal based on one or more journal conditions to more efficiently utilize system resources. The adaptive parameters used to adapt the optimization include aggressiveness parameters and the location of the optimizer, where the aggressiveness parameters specify the intensity of optimization of the journal by the journal optimizer. For example, the adaptive journal mechanism may dynamically adjust an adaptive parameter of the optimizer to increase optimization of the journal when the resource utilization indicates the resources are underutilized and decrease optimization of the journal when the resource utilization indicates resources are strained.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: Systems and methods are provided for reorganizing a partition-by-growth database with LOB columns. An example method includes creating a partition assignment table by performing a logical reorganization of the partition-by-growth database, wherein an assignment of at least one base table row and its corresponding LOB in the partition assignment table differs from a current assignment of the at least one base table row. The method also includes performing a base table reorganization based on the partition assignment table in parallel with a LOB auxiliary table reorganization based on the partition assignment table and a LOB dataset assignment table. The method may also include applying changes to the reorganized base tables and the reorganized LOB auxiliary tables using the partition assignment table.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: An application may store data to a dataset comprising a plurality of volumes stored on a plurality of storage systems. The application may request a dataset image of the dataset, the dataset image comprising a volume image of each volume of the dataset. A dataset image manager operates with a plurality of volume image managers in parallel to produce the dataset image, each volume image manager executing on a storage system. The plurality of volume image managers respond by performing requested operations and sending responses to the dataset image manager in parallel. Each volume image manager on a storage system may manage and produce a volume image for each volume of the dataset stored to the storage system. If a volume image for any volume of the dataset fails, or a timeout period expires, a cleanup procedure is performed to delete any successful volume images.

Abstract: Techniques are disclosed for resynchronizing a node of a distributed storage system with other nodes of the distributed storage system. Some embodiments presented herein include a computer-implemented method for resynchronizing a node of a distributed storage system with other nodes of the distributed storage system. The method comprises identifying an out-of-sync block of the node. The method further comprises determining that the out-of-sync block is a code block, wherein the code block is generated by performing an erasure coding operation on data blocks which are stored in the other nodes. The method further comprises locating a mirrored code block in an address space maintained for mirrored code blocks. The method further comprises storing contents of the mirrored code block in a storage location of the out-of-sync block.

Abstract: Data protection using change-based measurements in block-based backup is disclosed. Block change information indicating an extent of change associated with a volume may be determined. The block change information may be based at least in part on stored information indicating monitored changes to blocks in the volume. A backup operation may be initiated based at least in part on the determined block change information.

Abstract: A processor of an aspect includes a decode unit to decode a persistent store fence instruction. The processor also includes a memory subsystem module coupled with the decode unit. The memory subsystem module, in response to the persistent store fence instruction, is to ensure that a given data corresponding to the persistent store fence instruction is stored persistently in a persistent storage before data of all subsequent store instructions is stored persistently in the persistent storage. The subsequent store instructions occur after the persistent store fence instruction in original program order. Other processors, methods, systems, and articles of manufacture are also disclosed.

Abstract: Recovery of a database system by taking the database system offline is initiated. Thereafter, recovery operations specified by a redo log of the database system are replayed. A cleanup log is generated that identifies cleanup operations occurring during the replay of the recovery operations for garbage collection. Concurrent with the startup of the database, garbage collection of the cleanup operations as specified in the database savepoint is initiated. In addition, concurrent with the replay of the recovery operations, garbage collection of the cleanup operations specified by the cleanup log is initiated. The amount of parallelization of the garbage collection can be varied based on utilized processor resources. The database system is later brought online after all of the recovery operations are replayed.

Abstract: The present disclosure provides a method and an apparatus for a storage system. The storage system includes an active storage site and a standby storage site, and a logical unit number LUN for a storage pool of the active storage site is replicated to a LUN for a storage pool of the standby storage site to form a consistency group. The method comprises: creating a first LUN and a second LUN as mirrors of each other at the active storage site and the standby storage site, respectively; adding the mirrors formed by the first LUN and the second LUN to the consistency group; after the second LUN is added to the storage pool of the standby storage site, adding the first LUN to the storage pool of the active storage site.

Abstract: Reconstructing in-memory data block indices in a distributed data storage system where data blocks are stored in extents and the extents are replicated across storage devices. In one aspect, based on a reboot of a storage device and a copy of an extent stored in the storage device being in an open state, appends for data blocks in the copy of the extent stored in the storage device are replayed to reconstruct an in-memory data block index for the copy of the extent. In another aspect, based on a reboot of a storage device and a copy of an extent being in a closed state, a data block index for the copy of the extent is retrieved from non-volatile storage of the storage device and the retrieved data block index stored in memory at the storage device.

Abstract: Master-slave pairs can be used to provide data redundancy in an electronic data environment. A master peer can include a B-tree with references to the corresponding data. When provisioning a slave, the master can send a point-in-time copy of the B-tree to the slave, which can allocate the necessary space on local storage and update the references of the B-tree to point to the local storage for the slave. If the master and slave become disconnected, one of the peers can function as a solo master until the peers are again connected, at which point the old peer can be brought current or a new slave provisioned. A log peer can also be provisioned by a solo master, which can store data for operations received during the disconnect for use in catching up a slave peer, which could be the old slave, the log peer, or a new peer.

Abstract: Example embodiments of the present invention relate to a method, a system, and a computer program product for performing data replication and backup. The method comprises performing a first data replication of a production site storage to a replication site storage and performing a first backup of the production site storage to a production site backup storage. A second data replication then may be performed from the production site backup storage to a backup site backup storage.

Abstract: A computer program product, system, and method for generating coded fragments comprises intercepting, at a splitter, a write request from a host to storage, the write request comprising write data; sending the write request to a data protection appliance (DPA); calculating a probability of ransomware within the host; if the probability of ransomware is less than or equal to a first threshold, sending an acknowledgement (ACK) to the splitter; if the probability of ransomware is greater than a first threshold value and less than or equal to a second threshold value, creating a bookmark and sending an ACK to the splitter; and if the probability of ransomware is greater than the second threshold value, sending a delayed ACK to the splitter.

Abstract: Each of a plurality of database transactions are logged (i.e., recorded) in a log. Concurrent with the logging, one or more characteristics of the log are monitored. Thereafter, a savepoint is triggered when a pre-defined condition is met as indicated by the monitoring. The triggered savepoint can override or accelerate a savepoint that would have otherwise been triggered based on pre-specified parameters.

Abstract: Provided are a computer program product, system, and method for generating node access information for a transaction accessing nodes of a data set index. Pages in the memory are allocated to internal nodes and leaf nodes of a tree data structure representing all or a portion of a data set index for the data set. A transaction is processed with respect to the data set that involves accessing the internal and leaf nodes in the tree data structure, wherein the transaction comprises a read or write operation. Node access information is generated in transaction information, for accessed nodes comprising nodes in the tree data structure accessed as part of processing the transaction. The node access information includes a pointer to the page allocated to the accessed node prior to the transaction in response to the node being modified during the transaction.

Abstract: A database management system generates a log each transaction during execution of a plurality of transactions and stores the generated logs in log storage areas. The database management system records sequence numbers of logs at least in the generated logs of transactions belonging to a set of transactions of which results are different depending on a transaction execution order.

Abstract: An application may store data to a dataset comprising a plurality of volumes stored on a plurality of storage systems. The application may request a dataset image of the dataset, the dataset image comprising a volume image of each volume of the dataset. A dataset image manager operates with a plurality of volume image managers in parallel to produce the dataset image, each volume image manager executing on a storage system. The plurality of volume image managers respond by performing requested operations and sending responses to the dataset image manager in parallel. Each volume image manager on a storage system may manage and produce a volume image for each volume of the dataset stored to the storage system. If a volume image for any volume of the dataset fails, or a timeout period expires, a cleanup procedure is performed to delete any successful volume images.

Abstract: A computer-implemented method according to one embodiment includes identifying one or more accessible server nodes within a plurality of nodes of a failed clustered system, retrieving a plurality of fragments of configuration data from the one or more accessible server nodes, and constructing a backup state for the failed clustered system, utilizing the plurality of fragments of the configuration data.

Abstract: Disclosed aspects relate to using a database driver to manage application resiliency with respect to a shared pool of configurable computing resources. A transaction request having a set of command data is received. The set of command data for the transaction request is captured in a memory device coupled with the database driver. The transaction request is connected with a first asset of the shared pool of configurable computing resources. An error event is detected related to the first asset. It is ascertained that the error event indicates a configuration for an automatic client reroute (ACR) operation. Using the ACR operation, the transaction request is connected with a second asset of the shared pool of configurable computing resources. The set of command data for the transaction request is transmitted from the memory device coupled with the database driver to the second asset.

Abstract: The storage system includes storage devices which have plural storage blocks. A first node with a first processor memory is configured to store first information which manages states of the storage blocks, and a second node including a second processor and memory is configured to store second information which manages the states of the storage blocks are included in the storage system to handle file operations. When a failure occurs on the second node, the first processor is configured to change the state of one or more storage blocks, previously reserved by the second node to be used for storing data by the second node, to one or more garbage blocks and perform garbage resolution processing thereon before becoming available for data storage. This process allows for rapid recovery of the filesystem as the entirety of the filesystem metadata need not be examined.

Abstract: Upon receiving a medium storage request from a client apparatus, an information management server apparatus according to an embodiment evaluates, based on at least one policy file read out based on at least one parent management ID in the medium storage request and user information in the medium storage request, whether medium storage responding to the medium storage request is possible. Upon evaluating that the medium storage is possible, the information management server apparatus creates composite access control which do not violate any management rules included in the at least one policy file.

Abstract: The present disclosure relates generally to a data integration system that integrate an object based data integration tool, such a GUI-based data integration tools, with version control systems using a relational database repository for persistence. Examples of distributed version control systems include Git, Mercurial, and Bazaar, and examples of centralized version control systems include Subversion, CVS etc. in centralized or distributed environments.

Abstract: Disclosed herein are a system, non-transitory computer-readable medium, and method for maintaining storage unit transaction logs. Storage unit transaction logs received from controllers are separated.

Abstract: The present disclosure relates generally to a data integration system that integrate an object based data integration tool, such a GUI-based data integration tools, with version control systems using a relational database repository for persistence. Examples of distributed version control systems include Git, Mercurial, and Bazaar, and examples of centralized version control systems include Subversion, CVS etc. in centralized or distributed environments.

Abstract: This document relates to data storage techniques. One example can buffer write commands and cause the write commands to be committed to storage in flush epoch order. Another example can maintain a persistent log of write commands that are arranged in the persistent log in flush epoch order. Both examples may provide a prefix consistent state in the event of a crash.

Abstract: A backup computing device detects an interruption while receiving a backup copy of a transaction log of a primary database and directs a secondary computing device to continue generation of the backup copy of the transaction log, based on the copy of the transaction log of the primary database. A primary computing device directs a secondary computing device to generate a backup copy of the data file and the primary computing device generates a backup copy of the transaction log, of a primary database. A primary computing device accesses a snapshot of a plurality of snapshots of a primary database, and generates a first portion of a backup copy of the database. The primary computing device directs a second computing device to generate a non-overlapping portion of the backup copy of the database based on a second snapshot of the plurality of snapshots.

Abstract: Various embodiments for managing data integrity in a computing storage environment, by a processor device, are provided. In one embodiment, a method comprises applying a copy-on-write technique to a cache in a computer storage system such that each write arriving in the cache is assigned to a separate new physical location and registered sequentially in an order the write arrived, for preserving a state of the computer storage system during a failure event.

Abstract: A method for dynamically updating database statistics and a plurality of access paths associated with at least one database table is provided. The method may include identifying at least one dynamically changing column associated with the at least one database table. The method may further include determining at least one predicate pattern associated with the identified at least one dynamically changing column. Additionally, the method may include dynamically detecting at least one range change associated with the determined at least one predicate pattern based on at least one record insert. The method may also include updating the plurality of database statistics associated with the identified at least one dynamically changing column based on the dynamically detected at least one range change. The method may further include determining at least one filter factor based on the updated plurality of database statistics associated with the identified at least one dynamically changing column.

Abstract: Methods and systems for creating a compact tree node representation of an XML document. One implementation commences by allocating memory for storing an XML tree index data structure, then allocating another separate portion of memory to store a hash table. Then, traversing an XML document to process the traversed nodes as follows: (a) when the traversed node is an element node, then adding the element node to the XML tree index data structure (b) when the traversed node is a text node, then populating a text node index into the XML tree index data structure and copying the text node values to the hash table, and (c) when the traversed node is an attribute node, then populating an attribute node index into the XML tree index data structure. Such a structure supports fast index-based tree restructuring, and permits very large XML document to be accessed within tight memory size constraints.

Abstract: Partitioning of a source table of a database to a target table is initiated. Thereafter, a replay table is generated that is populated with triggers for database operations performed on the source table for subsequent replay for the target partitions. Data is later moved (e.g., asynchronously moved, etc.) from the source table to the target table. The database operations are replayed on the target table T subsequent to the moving of the data using the replay table. In addition, the source table is dropped when all of the data has been moved to the target table and there are no operations requiring replay. Related apparatus, systems, techniques and articles are also described.

Abstract: An in-vehicle terminal and a mobile terminal receive a request and a first identifier for identifying the request and store the first identifier in a buffer, the mobile terminal sends the request to a server, which stores the first identifier in a buffer when the request includes the first identifier and generates a second identifier when the request does not include the first identifier. When an event occurs, the mobile terminal generates and stores a message and a message code related to the second identifier together with the second identifier, transfers a response including the first identifier, the message and the message code to the mobile terminal when the first identifier exists in the buffer, and transfers a response including the message and the message code to the mobile terminal when the first identifier does not exist in the buffer. The mobile terminal displays the message and the message code.

Abstract: The efficient backing up of a hierarchical system in cloud blob storage. The hierarchical structure of the system as it existed at a prior instance in time is reconstructed. A change journal that represents changes in the file system that prior instant in time is then used to formulate an updated file system hierarchy as it exists at a second instant in time. An updated injected representation of the file system, and updated file system reversal information is then formulated and provided to cloud blob storage. The injected representation of the file system is a one-to-one function of the content of the file system, in that the reversal information can be used to recover the content of the file system. Injected representations of various nodes in the system hierarchy may also be remotely stored.

Abstract: Disclosed herein are system, method, and computer program product embodiments for stream optimized data processing. An embodiment operates by receive a stream of data in a streaming data format. A query associated both with the stream of data and one or more records of a database is determined. It is determined whether the one or more records of the database are stored in a local cache. Those records not stored in the local cache are retrieved from the database and converted into the streaming data format. A query response, including references to each of the one or more records stored in the local cache in the streaming data format, is provided for execution of the query.

Abstract: A transaction engine includes a multi-version concurrency control (MVCC) module that accesses a latch-free hash table that includes respective hash table entries that include respective buckets of respective bucket items. The bucket items represent respective records, the respective bucket items each including a value indicating a temporal most recent read time of the item and a version list of descriptions that describe respective versions of the respective records, the MVCC module performing timestamp order concurrency control, using the latch-free hash table. Recovery log buffers may be used as cache storage for the transaction engine.