Abstract: In a Byzantine computing environment in which a database is sharded or partitioned among multiple clusters of computing nodes, consensus for and execution of data transactions (e.g., transactions that require and/or affect data of one or more shards) are achieved in a resilient manner. Within some clusters, multiple primary replicas concurrently propose transactions for processing in parallel by all replicas. For some multi-shard transactions, shards involved in the transactions may be logically ring-ordered; each shard in turn achieves consensus among its nodes to commit the transactions, and then executes its portion of the operation after consensus is obtained among all shards. For some other multi-shard transactions, involved shards first determine whether local data constraints are satisfied, after which data modifications are made in parallel.

Abstract: A transaction processing protocol for serverless database management systems can use a transaction scheduler to guarantee consistent serializable execution though analysis of the access pattern of transaction types and appropriate ordering of the transaction's events at runtime. A transaction topology is determined for each type of transaction and these are combined and used to generate a serialization graph. Cycles in the serialization graph are identified and breaking transaction types which may break the cycles are determined. When transaction requests are received, a breaking type of transaction is scheduled as a last transaction in the current epoch and later transactions not having the breaking transaction type are scheduled to execute in the next epoch.

Abstract: This application discloses a data replication method and apparatus, a computer device, and a storage medium, and belongs to the field of database technologies. The method includes: adding, when a commitment operation of a transaction is detected, historical state data of the transaction to a data queue; adding at least one piece of historical state data in the data queue to a transmission buffer; and replicating, when a first preset condition is met, the at least one piece of historical state data in the transmission buffer to a cluster device. In this application, a node device does not need to convert an original historical state data format into a log format, and the cluster device does not need to parse a log into an original data format for storage, thereby avoiding a cumbersome log replay procedure, and improving the efficiency of the data replication process.

Abstract: A method, apparatus, and computer program product for executing a relational database management system (RDBMS) in a computer system, wherein the RDBMS manages a relational database comprised of at least one column-partitioned base table storing data. Column values from at least one column of the column-partitioned base table are stored in one or more containers spread across one or more data blocks. Metadata comprising summarized information about the column values in the containers is stored in a metadata index subtable. A query with a filtering condition on the column is applied to the metadata index subtable before the column-partitioned base table is accessed, so that only qualified containers and data blocks are accessed, and unqualified containers and data blocks are eliminated, when responding to the query.

Abstract: Methods, computer program products, computer systems, and the like are disclosed that provide for scalable deduplication in an efficient and effective manner. For example, such methods, computer program products, and computer systems can include determining, at a source site, whether metadata has been received from a target site, and, in response to a determination that the metadata has been received at the source site, retrieving the at least one unit of the source data from the source data store using the metadata and sending, from the source site, the at least one unit of source data to the target site.

Abstract: Techniques are provided for utilizing a log to free pages from persistent memory. A log is maintained to comprise a list of page block numbers of pages within persistent memory of a node to free. A page block number, of a page, within the log is identified for processing. A reference count, corresponding to a number of references to the page block number, is identified. In response to the reference count being greater than 1, the reference count is decremented and the page block number is removed from the log. In response to the reference count being 1, the page is freed from the persistent memory and the page block number is removed from the log.

Abstract: Methods, systems, and computer-readable media for automatic archiving of data store log data are disclosed. One or more operation records in a log are selected for archival. The one or more operation records comprise data indicative of operations performed on one or more data objects of a data store. The one or more operation records are selected for archival prior to deletion from the log. The one or more operation records are replicated from the log to an archive. Based at least in part on the replicating, the one or more operation records in the log are marked as archived. Based at least in part on the marking as archived, the deletion of the one or more operation records from the log is permitted.

Abstract: A node includes storage that stores a portion of high accessibility data corresponding to a shard that is divided into revisions. The portion of the high accessibility data is managed based on an activity state model including a writing state, a resolving state, a metadata update state, and a reading state. The node further includes a repository manager that obtains a cross-shard read request for the high accessibility data; makes a determination that the portion of the high accessibility data is not in the reading state; in response to the determination, adds the cross-shard read request to a delayed action queue at the revision of a last delayed action in the delayed action queue; and after processing the last delayed action in the delayed action queue and while in the resolving state: converts the cross-shard read request to a read request; and processes the read request.

Abstract: Computing platforms, methods, and storage media for data movement are disclosed. Exemplary implementations may: obtain a data transfer command including a table-based characterization of a set of data to be transferred from an on-premise platform to a cloud platform, the data transfer command further including a date specification; automatically determine, based on the table-based characterization of the data to be transferred and based on the date specification, a set of files to be transferred; and initiate transfer of the set of files from the on-premise platform to the cloud platform. In an implementation, data is moved at the table level, and the files associated with the specified tables/date range are automatically determined. A single command, which may reference a CSV file, may be used to efficiently and reliably transfer a large amount of data without a user having to specify the specific files to be transferred.

Abstract: This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize a plurality of data validation processes to merge or replicate data across databases for downstream operations. For example, in response to a request to merge data from one or more source servers to one or more destination servers, a disclosed system determines database events to merge based on a plurality of database tables accessed in connection with a card account transaction. The disclosed system validates subsets of database events for the accessed database tables based on the quantities and event identifier sequencing of the database events. Additionally, the disclosed system replicates the database events from the source servers to the destination servers in response to validating the completeness and ordering of the database events for the card account transaction.

Abstract: The present disclosure involves systems, software, and computer implemented methods for aligned purpose disassociation in a multi-system landscape. One example method includes receiving, from multiple systems, a can-disassociate status for a purpose for an object instance. The status from a respective system can be an affirmative status that indicates that the system can disassociate the purpose from the instance or a negative status that indicates that the system cannot disassociate the purpose from the instance. The received statuses are evaluated to determine a central disassociate purpose decision for the purpose for the instance. The central disassociate purpose decision can be to disassociate the purpose from the instance when no system has the negative status and to not disassociate the purpose from the instance when at least one system has the negative status. The central disassociate purpose decision is provided to at least some of the multiple systems.

Abstract: An automated method is provided for use when replacing a currently operating data replication engine in a first system with a new data replication engine in the first system in a bidirectional data replication environment. The currently operating data replication engine in the first system and the new data replication engine in the first system replicates first database transactions from an audit trail of a first database in the first system to a second database in a second system. The new data replication engine in the first system generating a list of active database transactions in the first system, and sends the list of active database transactions to the new data replication engine in the second system as a first token.

Abstract: A computer implemented method and system for a selectively replicated trustless persistent store is provided using a bilateral distributed ledger. The selectively replicated trustless persistent store synchronizes current state data stores shared among multiple parties. Data modifications may be made in any shared store locally and then are automatically replicated across other permissioned stores. The selectively replicated trustless persistent store is responsible for getting the data validated and agreed upon before committing locally.

Abstract: The invention provides for a cloud-based solution that saves all the data in the cloud storage. The peer devices synchronize data among each other independent of the operating system since the data is synced via web services. Synchronization of data among peer devices is possible even when cloud service is unavailable via a router, Wi-Fi, Bluetooth, NFC or any other mechanism. The peer devices form a hierarchical structure, which designates a master, and the master communicates with the cloud-based service to synchronize data. The master then synchronizes data with the other peer devices in the hierarchy. New devices can be added to the peer devices and can join the hierarchy.

Abstract: Systems and devices can include a controller and a command queue to buffer incoming write requests into the device. The controller can receive, from a client across a link, a non-posted write request (e.g., a deferred memory write (DMWr) request) in a transaction layer packet (TLP) to the command queue; determine that the command queue can accept the DMWr request; identify, from the TLP, a successful completion (SC) message that indicates that the DMWr request was accepted into the command queue; and transmit, to the client across the link, the SC message that indicates that the DMWr request was accepted into the command queue. The controller can receive a second DMWr request in a second TLP; determine that the command queue is full; and transmit a memory request retry status (MRS) message to be transmitted to the client in response to the command queue being full.

Abstract: Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: A method, system and computer program product for copying data from a source database to a target database. A first database engine of the source database assigns a data-change-ID to each changed row in the source table. The data rows of the source table are stored in a plurality of source pages. Each source page comprises a source-page-ID and a highest-data-change-ID indicating its most recently changed data row. The target table comprises a source-page-ID-column. A second database engine of the target database maintains a metadata table comprising a source-page-ID column and a highest-data-change-ID column. The highest-data-change-IDs in the metadata table are compared with the highest-data-change-IDs of the source pages for selectively copying source table rows of source pages whose highest-data-change-ID is higher than their highest-data-change-ID in the metadata table to the target table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Abstract: Transaction recovery in a cloud computing environment. Transaction recovery includes determining whether a first container of a plurality of containers in the cloud computing environment is crashed. In response to the first container being crashed, the method may include determining at least one transaction log entry from a first log file corresponding to the first container. Transaction recovery includes retrieving at least one in-doubt transaction corresponding to the first container from a first database corresponding to the first container. Transaction recovery includes comparing the at least one transaction log entry with the at least one in-doubt transaction for transaction recovery.

Abstract: Disclosed embodiments are related to Master Data Management (MDM) technologies. Each DB entity (record) in component databases (DB) is represented as a judgment, and an MDM system unifies judgments obtained from the component DBs into a unified set of judgments. In the unified set of judgments, linkages are judgments asserting that particular DB entities from different DBs are the same/similar, and a golden record comprises field values describing each of the DB entities. In making judgments, the MDM system consolidates judgments (or records) describing the same subject into a single judgment, and performs updates to field values in a manner that does not violate referential integrity. Each update is associated with an MDM consistent state. Updates in the form of judgments are provided to the relevant component DBs, which are converted into serializable transactions associated with respective MDM consistent states. Other embodiments may be described and/or claimed.

Abstract: An augmented sleepable read-copy update implementation (PREEMPT_SRCU) combines elements of a tree-based sleepable read-copy update environment (Tree-SRCU) with elements of a preemptible read-copy update environment (Preemptible-RCU). The elements of Tree-SRCU may be used to manage PREEMPT_SRCU grace periods and handle PREEMPT_SRCU callbacks. The elements of Preemptible-RCU may be used to drive existing PREEMPT_SRCU grace periods to completion.

Abstract: The present disclosure provides a method and a device for data processing. The method includes acquiring at least two pathways of communication messages, where the at least two pathways of communication messages are messages intercepted in a bypass manner from messages transmitted by a service processing system to an external system; and the service processing system does not execute logic of record storage; processing the at least two pathways of communication messages, and determining communication messages to-be-stored from the at least two pathways of communication messages processed; and according to the at least two pathways of communication messages, storing the communication messages to-be-stored in a database.

Abstract: A method is provided for performing transaction processing in a system that includes a plurality of nodes. Each of the nodes include a processor, an application executing in the processor, and an instance of a database used by the application. Each node executes transactions. At least two of the nodes are selected to receive the same request to process a transaction, and the processor at each of the at least two nodes executes the same request to process a transaction. Each of the at least two nodes replicates the steps and operations of the transactions to one or more other nodes in the system. The processor at the one or more other nodes compares the replicated steps and operations of the transactions that were received from each of the at least two nodes and executes one or more remediation actions when the result of the comparison indicates that the replicated steps and operations of the replicated transactions do not match each other.

Abstract: A system includes a multi-layer capacity configuration optimization (CCO) stack to generate a token containing prescriptions for optimize capacity configuration of a database container in a NoSQL database cloud service. The system may aggregate the capacity utilization data; predict, based on the aggregated capacity utilization data, respective prediction-based processing capacity utilizations for the database container; determine a target processing capacity utilization value from the prediction-based processing capacity utilizations; calculate respective provisioned processing capacity utilizations based on the target processing capacity utilization value; evaluate a consumption metric based on the prediction-based processing capacity utilizations and the provisioned processing capacity utilizations; select one of the predetermined capacity modes as a recommended capacity mode for the database container based on the consumption metric.

Abstract: An embodiment includes extracting, after a first specified time interval, recovery data from a recovery log for a data table corresponding to a set of database operations on the data table that occurred during the first specified time interval. The embodiment also generates index data for a progress record associated with the first time interval. The progress record includes a first data field for storing data representative of a first net change to the data table resulting from a first subset of the set of database operations. The first subset of database operations are associated with a first transaction. The embodiment also detects a commit operation in the first subset of database operations that committed the first transaction. The embodiment also recovers the first transaction in a target data table using the recovery log and the first net change data associated with the first transaction in the progress record.

Abstract: Offline building of a projected data subset may be performed. A request to create a data set that is a projected subset of data from a source data set may be received. A data store separate from the data store storing the source data set may store a copy of the source data set that is used to replicate items to the projected subset of data according to a schema for the projected data subset. Updates made to the source data set may also be replicated to the projected data subset according to the schema. Conflicts between replicated items and replicated updates to the projected data set may be resolved by comparing a version identifier for the replicated update and replicated item to determine what to store in the projected data subset.

Abstract: A streaming distributed decentralized database task system can perform multiple tasks on clusters of nodes without overloading the clusters' computational resources, such as disk, memory, processors, and network bandwidth. A cluster master can manage a job and add items to node queues. A node manager accepts or rejects queue items based on streaming task limits that are applied at the node level.

Abstract: Systems, methods, and software for management of partitioned data storage spaces is provided herein. An exemplary method includes storing sets of structured data records among partitioned data storage spaces, with data fields of the structured data records correlated among the sets by relational associations. The method includes, in a data center that receives change actions related to the structured data records, selectively placing the change actions into a plurality of change feeds, where the change feeds can be commutatively executed in parallel to implement the change actions. The method further includes implementing the change actions and propagating replication data comprising the change actions in the plurality of change feeds.

Abstract: Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.

Abstract: Disclosed is a computer program stored in a computer readable storage medium, which includes encoded commands, wherein executing the computer program by one or more processors of a computer system allows the one or more processors to perform steps for change data capture (CDC) between a source database and a target database.

Abstract: A system synchronizes change-data in a multi-tenant system with one or more external service provider systems.

Abstract: Systems, methods, and devices for storing database data in journal tables comprising a snapshot and a log table. A method includes defining a journal table comprising a snapshot and a log table, the snapshot comprising an up-to-date representation of data in the journal table at a point in time. The method includes assigning a timestamp to the snapshot indicating when the snapshot was generated. The method includes receiving a request to execute a transaction on the journal table to modify the data in the journal table, the transaction comprising one or more of an insert, a delete, an update, or a merge. The method includes inserting a new row into the log table in lieu of executing the transaction on the snapshot of the journal table, the new row comprising an indication of a change requested to be made to the journal table based on the transaction.

Abstract: A computer implemented method and system for a selectively replicated trustless persistent store is provided using a bilateral distributed ledger. The selectively replicated trustless persistent store synchronizes current state data stores shared among multiple parties. Data modifications may be made in any shared store locally and then are automatically replicated across other permissioned stores. The selectively replicated trustless persistent store is responsible for getting the data validated and agreed upon before committing locally.

Abstract: A streaming distributed decentralized database task system can perform multiple tasks on clusters of nodes without overloading the clusters' computational resources, such as disk, memory, processors, and network bandwidth. A cluster master can manage a job and add items to node queues. A node manager accepts or rejects queue items based on streaming task limits that are applied at the node level.

Abstract: Technologies are described for performing synchronization within a database environment. A source host stores multiple database tables and a replica host stores copies of the multiple database tables. During synchronization, replication log replay is blocked at the replica node, and a multithreaded process locks the multiple database tables of the source hosts, reactivates replication log generation for each of the multiple database tables, and then unlocks the database tables. A synchronization timestamp is acquired and used to synchronize the multiple database tables with the copies of the multiple database tables. After synchronization, replication log replay is reactivated at the replica host for the copies of the multiple database tables.

Abstract: A database session in an active standby server on which an active standby database resides receives a DML statement. The session is suspended while the statement is redirected over a database link to a primary database on which the statement is executed. Information associated with execution of the statement is communicated to the session in the active standby server. Redo records describing changes to the contents of the primary database are applied to the active standby database and control is returned to the session. Prior to commitment of a transaction including the statement, a query directed to data to which the statement was directed is received at the active standby server from a client and executed on the active standby database absent use of a database link based on whether information associated with a database session associated with the client matches the information associated with execution of the statement.

Abstract: Disclosed herein are system, method, and computer program product embodiments for a database recovery and optimization with batch processing system. An embodiment operates by retrieving a database log that includes a plurality of operations for modifying data of a database stored across a plurality of tables. From the database log, a plurality of consecutive insert operations for inserting data into the database are identified. The consecutive insert operations are sorted by table. The sorted insert operations are grouped into a batch message. The batch message is transmit to the database for replay. An acknowledgement is received that the replay has completed.

Abstract: Optimized location information is provided for a location based service. An event is detected which is indicative of an error in the location based service which degrades the performance of the location based service. A cause of the error is determined, and resolution of the error is initiated based on the determined cause.

Abstract: In accordance with embodiments, there are provided techniques for implementing batch processing in a multi-tenant database system. These techniques for implementing batch processing in a multi-tenant database system may enable embodiments to provide great flexibility to a tenant of the architecture to perform desired functions on content of the database while allowing the owner of the architecture schedule the processing of the functions on the content.

Abstract: A request to store a new value to a data entity to be fault tolerant across a plurality of nodes is received. The data entity being in an intermediate state is indicating to at least one less than a majority of the plurality of nodes. A request to read a stored value of the data entity is received. A latest value instance of the data entity among a majority nodes of the plurality of nodes is determined. A value of the latest value instance in a majority nodes of the plurality of nodes is stored as a next value instance of the data entity.

Abstract: Systems and method of the present invention are directed to data transfer systems and methods. Embodiments enable a user to access an electronic file to ascertain certain attribute data associated with one or more files, and allow the user to modify certain elements of the subject attributes file during the course of transferring the one or more files from a source location to a target or destination location.

Abstract: A funnel locking technique for normal read-copy update (RCU) grace period requests. Based on a calculated future normal RCU grace period, a traversal is initiated of a funnel lock embodied as a hierarchical tree of nodes. The funnel-lock traversal starts at an initial node whose lock is held throughout the funnel-lock traversal. For each node accessed during the funnel-lock traversal that is not the initial node, a lock on the node is held while accessing that node. For each accessed node, the funnel-lock traversal is terminated if the future normal RCU grace period has already been requested at that node, if the node believes that the future normal RCU grace period as already started, or if the node is not the initial node and believes that any normal RCU grace period is underway. Otherwise, a request for the future normal RCU grace period is recorded at the node.

Abstract: A first storage system is configured to participate in a replication process with a second storage system. The first and second storage systems initially operate in an asynchronous replication mode and subsequently transition to a synchronous replication mode. The first and second storage systems concurrently operate in both modes for at least a portion of the transition. While the first and second storage systems are concurrently operating in both modes, the first storage system is configured to detect a given data page that has been subject to one or more write operations in the first storage system since generation of a corresponding set of one or more snapshots for a most recent data transfer cycle of the asynchronous replication mode, to initiate a data transfer request for the given data page, and to update a content-based signature of the given data page. The second storage system determines whether to accept or reject the transfer of the given data page.

Abstract: A system and method for increased ability to share and interact with data objects spread across a cloud in a collaborative environment. There has thus been outlined, rather broadly, certain embodiments of the invention in order that the detailed description thereof herein may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional embodiments of the invention that will be described below and which will form the subject matter of the claims appended hereto.

Abstract: There is provided a data distribution storing method including: recognizing, by a cluster manager managing a cluster of a plurality of nodes, a vehicle storage installed in a vehicle; transmitting, by the cluster manager, a vehicle index request including identification information about the vehicle storage to an index server; receiving, by the cluster manager, a first vehicle index including information about a first time section during which first vehicle data is recorded in the vehicle storage from the index server; selecting, by the cluster manager, a node from the cluster; transmitting, by the vehicle storage, second vehicle data recorded in the vehicle storage during a second time section which is after the first time section, to the node; generating, by the cluster manager, a second vehicle index based on the first vehicle index and information about the second time section and the node.

Abstract: There is provided a data distribution storing method including: recognizing, by a cluster manager managing a cluster of a plurality of nodes, a vehicle storage installed in a vehicle; transmitting, by the cluster manager, a vehicle index request including identification information about the vehicle storage to an index server; receiving, by the cluster manager, a first vehicle index including information about a first time section during which first vehicle data is recorded in the vehicle storage from the index server; selecting, by the cluster manager, a node from the cluster; transmitting, by the vehicle storage, second vehicle data recorded in the vehicle storage during a second time section which is after the first time section, to the node; generating, by the cluster manager, a second vehicle index based on the first vehicle index and information about the second time section and the node.

Abstract: A cloud computing system includes a plurality of small medium business (SMB) network system and a cloud hosting and storage system. Each of the SMB network systems includes a plurality of first client terminals and a physical network-attached storage (NAS) configured to store one or more original files accessible by at least one of the first client terminals. The cloud hosting and storage system is configured to generate a virtual NAS corresponding to the physical NAS in accordance with one or more requests for cloud hosting service made by the physical NAS through a network, and allocate a virtual storage space for the SMB network system. The virtual storage space is configured to store one or more backup files of the original files stored in the physical NAS.

Abstract: A method is provided to verify the computational results of a transaction processing system that includes a parent node a plurality of child nodes. The parent node sends to at least two child nodes an identical request to process a transaction. The transaction is allowed to modify an application's state only if the validity of the result of the processing of the transaction is verified across participating child nodes. Otherwise, the transaction is aborted.

Abstract: A lower-level master node sends, to a higher-level master node, a table-based write set as a bundle of combinations of a shadow copy of the database and a heap tuple map expanded in a memory, and the higher-level master node that received the write set verifies whether the update has already been executed and sends the record of this update to the lower-level master node as a transaction log including a table number, whereby the database can be updated efficiently and consistently from the lower-level master node to the higher-level master node, and from the higher-level master node to the lower-level master node under its control.