Abstract: A method and a server system for creating consistent snapshots in a distributed storage system are disclosed. The server system stores entities in databases that includes a production portion for storing current versions of entities and a backup portion for storing backup versions of modified entities, receives a request for a snapshot of a portion of the databases at a predefined snapshot time. In response to the request, the server system generates a snapshot of the databases, the snapshot including: a backup version of an entity retrieved from the backup portion and a current version of another entity retrieved from the databases and produces a response to the request based on the snapshot. When there are several backup portions, a backup portion may be used separately or in conjunction with other backup portions to provide required backup versions to generate a snapshot.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: A method and a server system for creating consistent snapshots in a distributed storage system are disclosed. The server system stores entities in databases that includes a production portion for storing current versions of entities and a backup portion for storing backup versions of modified entities, receives a request for a snapshot of a portion of the databases at a predefined snapshot time. In response to the request, the server system generates a snapshot of the databases, the snapshot including: a backup version of an entity retrieved from the backup portion and a current version of another entity retrieved from the databases and produces a response to the request based on the snapshot. When there are several backup portions, a backup portion may be used separately or in conjunction with other backup portions to provide required backup versions to generate a snapshot.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: A server system having one or more processors and memory sends, to a first database, a first request for data based on first data-retrieval criteria. The first database is schemaless and operates in accordance with first database processing capabilities. The server system receives, in response to the first request, a first set of data from the first database, where the first set of data is selected by the first database in accordance with the first data-retrieval criteria. The server system generates a second database from the first set of data selectively retrieved from the first database. The second database has an associated schema and operates in accordance with second database processing capabilities different from the first database processing capabilities. The server system provides access to the first set of data in the second database in accordance with the second database processing capabilities.

Abstract: A server system having one or more processors and memory stores a plurality of entities in one or more databases, where the one or more databases do not guarantee strong consistency of the stored entities. The server system executes a first query against at least a first portion of the one or more databases to generate a first set of results that is not guaranteed to be strongly consistent. The server system also executes a second query, where executing the second query includes performing a consistency operation, the consistency operation guaranteeing that a second portion of the one or more databases is strongly consistent. After performing the consistency operation, the server system generates a second set of results from the second portion and merges the first set of one or more results and the second set of one or more results.

Abstract: A server system having one or more processors and memory receives, from a requestor, a select distinct query. In response to the query, the server system identifies a set of index portions according to requestor-specified filter criteria. The server system identifies one or more distinct value combinations for entities satisfying the filter criteria, including: obtaining an initial value of a query cursor; retrieving candidate value combinations; comparing candidate value combinations to identify distinct value combinations, if any, in accordance with the query; and updating the current value of the query cursor. The server system repeats the retrieving candidate value combinations and the comparing candidate value combinations one or more times for the set of index portions until at least a respective distinct value combination has been identified and transmits information corresponding to the respective distinct value combination to the requestor.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: Methods, systems, and apparatus are described for processing mutations for a remote database. In one aspect, a method includes receiving a log of database mutations from a client device, the log of database mutations indicating changes previously made to a client version of a database stored on the client device, and each database mutation specifying: an entity included in the database; an operation for the entity; and a client version number for the entity; identifying, in the log of database mutations, two or more database mutations that each specify a same client version number for a same entity; determining, based on a conflict resolution policy, that the two or more database mutations are eligible for collapsing; and collapsing the two or more database mutations into a single database mutation.

Abstract: A method and a test system for specifying a predetermined degree of inconsistency for test data are disclosed. The test system obtains a test policy, which specifies a predetermined degree of inconsistency between write operations and subsequent read operations on a set of data and subsequently receives a request to provide test data to an application. In response to the request to provide test data to the application the test system generates a set of test data including a plurality of entities retrieved from the set of data, based at least in part on the test policy. The test data includes a respective entity that is not consistent with a previous write operation. The test system further provides the set of test data to the application. The application optionally processes the set of test data to produce results, which are used to determine performance of the application.

Abstract: A server system having one or more processors and memory receives a first query from a requestor and obtains a first query-processing plan including a first plurality of query-processing steps for processing the query. Supported query-processing steps comprise query-processing steps in a set of supported query-processing steps, and unsupported query-processing steps comprise query-processing steps not in the set of supported query-processing steps. In accordance with a determination that the first plurality of query-processing steps includes one or more of the supported query-processing steps and one or more unsupported query-processing steps including a respective unsupported query-processing step, the server system performs the one or more supported query-processing steps to produce intermediate results and generates a first response to the first query that includes the intermediate results and a representation of the respective unsupported query-processing step.

Abstract: A search server system having one or more processors and memory receives a search query including filter criteria. The filter criteria specify operands and a logical relationship. In response to the search query the server identifies index portions sorted in a same sort order. The server identifies matching entities that match the search query by identifying index entries having a first identifier and matching filter(s) of a first operand and identifying, index entries having a second identifier that is sequentially adjacent to the first identifier in the sort order and matching filter(s) of a second operand. When index portion(s) associated with the first operand include index entries having a respective identifier between the first identifier and the second identifier and matching the filter(s) of the first operand, the server marks a respective entity associated with the respective identifier as matching the search query and sends search results to the client.

Abstract: A server system having one or more processors and memory receives a first request from an application and processes at least a portion of the first request in accordance with a predefined set of resource constraints. In accordance with a determination that the predefined set of resource constraints has been met prior to completing processing the first request, the server system terminates processing of the first request prior to completing processing of the first request, generates a first intermediate result; and provides a response to the application. The response includes the first intermediate result, and request status information enabling the application to request resumption of the first request.

Abstract: A computer system having memory and one or more processors running an untrusted internal process determines whether the first interaction meets a first trigger condition and when the first interaction meets the first trigger condition, the computer system generates a first marker for the first data item. The computer system requests that a trusted external process perform the first interaction with the first data item and store the first marker in the database. After the first data item is stored, the computer system requests markers stored in the database and receives the first marker. The computer system determines whether an operation corresponding to the first data item is to be executed based on the first marker. When the computer system determines that a first operation corresponding to the first data item is to be executed, the computer system requests execution of the first operation.

Abstract: A server system having one or more processors and memory launches an instance of an application subject to a first set of resource constraints. Exceeding the first set of resource constraints will cause the application instance to be terminated. The server system transmits a first request from the application instance to a datastore server system. The first request is subject to a second set of resource constraints that are more restrictive than the first set of resource constraints. The server system receives a first response to the first request. When processing the first request exceeds the second set of resource constraints, the first response includes a first set of partial results and information enabling the application instance to request resumption of processing the first request, and when the first request was processed without exceeding the second set of resource constraints, the first response includes complete results.

Abstract: A server system having memory and one or more processors running a trusted internal process receives, from an untrusted external process of one or more external processes, a request to interact with a data item. In response to the request, the server system interacts with the first data item and stores a marker indicative of the interaction with the data item. After storing the marker, the server system identifies a plurality of markers corresponding to a plurality of data items and retrieves information corresponding to the plurality of markers. The retrieved information enables a determination, for each data item in the plurality of data items, as to whether an operation corresponding to the data item is to be executed. When a determination has been made that an operation corresponding to the data item is to be executed, the server system deletes the markers in conjunction with execution of the operation.

Abstract: A search server system having one or more processors and memory receives a search query including filter criteria. The filter criteria specify operands and a logical relationship. In response to the search query the server identifies index portions sorted in a same sort order. The server identifies matching entities that match the search query by identifying index entries having a first identifier and matching filter(s) of a first operand and identifying, index entries having a second identifier that is sequentially adjacent to the first identifier in the sort order and matching filter(s) of a second operand. When index portion(s) associated with the first operand include index entries having a respective identifier between the first identifier and the second identifier and matching the filter(s) of the first operand, the server marks a respective entity associated with the respective identifier as matching the search query and sends search results to the client.

Abstract: A server system having one or more processors and memory stores a plurality of entities in a schemaless database. The entities are not structured in accordance with a predefined schema. The server system generates an index for the plurality of entities stored in the schemaless database. The index has a plurality of index entries sorted in a sort order. The server system receives a first request from an application and, in response to the first request, accesses an empirically-determined schema and generates a first response based on the empirically-determined schema. The empirically-determined schema is generated from the index. After generating the first response, the server system sends the first response to the application.

Abstract: A search server system having one or more processors and memory receives a search query including filter criteria. The filter criteria specify operands and a logical relationship. In response to the search query the server identifies index portions sorted in a same sort order. The server identifies matching entities that match the search query by identifying index entries having a first identifier and matching filter(s) of a first operand and identifying, index entries having a second identifier that is sequentially adjacent to the first identifier in the sort order and matching filter(s) of a second operand. When index portion(s) associated with the first operand include index entries having a respective identifier between the first identifier and the second identifier and matching the filter(s) of the first operand, the server marks a respective entity associated with the respective identifier as matching the search query and sends search results to the client.