Abstract: In some implementations, the techniques described herein relate to a method including: receiving, by a processor, a query from a client device; distributing, by the processor, the query to a plurality of shards; receiving, by the processor, a plurality of array provider data structures from the plurality of shards, a given array provider data structure identifying responsive identifiers from a corresponding shard; materializing, by the processor, the plurality of array provider data structures; persisting, by the processor, a portion of responsive data on disk while materializing the plurality of array provider data structures; merging, by the processor, data stored on the disk; and returning, by the processor, a result set based on the data to the client device.

Abstract: A system for indexing includes an interface and a processor. The interface is configured to receive a metadata amendment. The metadata amendment defines a change to a transactional index schema to a first transactional index. The first transactional index comprises a querying system data. The processor is configured to maintain a first state of the first transactional index. The first transactional index does not incorporate the metadata amendment. The processor is configured to build a second transactional index incorporating the metadata amendment, determine whether an indication to switch from the first transactional index to the second transactional index has been received, and indicate to switch from using the first transactional index to the second transactional index for queries regarding a system data.

Abstract: A system for throttling includes an interface and a processor. The interface is configured to receive a read task. The processor is configured to determine a read engine of a set of read engines for the read task, submit the read task to the read engine for execution, determine whether the read task execution time exceeds a resubmit threshold time, and in the event that the read task execution time exceeds the resubmit threshold time, resubmit the read task, wherein resubmitting the read task includes determining a new read engine for the read task.