Abstract: A consistency verification system that verifies data consistency in a content item management system. The system processes a write request. The write request includes a set of preconditions, a key, and data to write in a write operation. Processing the write request includes determining that the set of conditions are met, committing the write request at a commit time, and assigning a timestamp indicating the commit time to the write request. The system verifies data consistency associated with the write request. The verifying data consistency includes sending a replay request corresponding to the write request. The system determines whether results of the replay are consistent with results of the write request. Responsive to determining whether results of the replay are consistent with results of the write request, the system outputs information describing data inconsistency.

Abstract: A consistency verification system that verifies data consistency in a content item management system. The system maintains a plurality of verifiers for checking data consistency, each verifier comprising instructions for verifying consistency for a type of requests. The system may verify the original request by selecting a verifier from the plurality of verifiers based on a type of the original request. Using the selected verifier, the consistency verification system may send verification requests to the content item management system at a second timestamp that is a period of time after the original timestamp. The system may determine whether the second response is consistent with the first response. If the responses are not consistent, the system may output information describing data inconsistency. If the responses are consistent, the system may output results and perform further verifications to help ensure data accuracy.

Abstract: A consistency verification system that verifies data consistency in a content item management system. The system maintains a plurality of verifiers for checking data consistency, each verifier comprising instructions for verifying consistency for a type of requests. The system may verify the original request by selecting a verifier from the plurality of verifiers based on a type of the original request. Using the selected verifier, the consistency verification system may send verification requests to the content item management system at a second timestamp that is a period of time after the original timestamp. The system may determine whether the second response is consistent with the first response. If the responses are not consistent, the system may output information describing data inconsistency. If the responses are consistent, the system may output results and perform further verifications to help ensure data accuracy.

Abstract: A content management system stores distributed data tables containing key-value pairs across a plurality of nodes and maintains a plurality of slices, with each slice corresponding to a contiguous key range across the data tables. The content management system may rebalance data among nodes by performing operations such as transferring, merging, or splitting slices. Each operation may be accomplished by performing multiple actions, and each action may cause change in states for slices. During an operation, slices may go through a series of state transitions. For each state transition, the content management system may record a timestamp when the state transition took place and the content management system may maintain a log that records timestamped state transitions associated with slices. The content management system may also perform various invariant checks and determine whether to reject or allow a state transition based on results of invariant checks.