Abstract: Methods and systems for shard optimized database queries using parameter-based indexes are provided. Exemplary methods include: receiving a database query that includes an index parameter and an index parameter range. A parameter table is accessed that contains an association between the parameter and parameter range and a shard identifier. Based on the parameter type and the range identified in the query, the relevant shards are identified, and the database query is limited to processing these shards.

Abstract: An automated system for allocation of resources in a cluster configured to run a search engine is disclosed. At least one master node includes a processing system. The processing system is configured to analyze the cluster based on measurements of different parameters. The results of the analysis can be used to allocate or reallocate the shards, allocate or reconfigure the workload portions assigned to the shards, and allocate or reconfigure the shards selectively to maintain high performance. Periodic analyses can predict future behavior, and reconciliations toward a target allocation can occur regularly to maximize system efficiency and performance.

Abstract: Provided are methods and systems for peer recovery from remote storage. An example method includes storing, to a remote storage, a data snapshot of a plurality of nodes of a cluster, determining, by the cluster, that a piece of data stored on a first node of the cluster needs to be copied to a second node of the cluster, and in response to the determination, causing, by the cluster, the second node to download a copy of the piece of data from the data snapshot in the remote storage, determining, by the cluster, that the piece of data stored on the first node differs from the copy of the piece of data stored on the data snapshot, and in response to the determination, causing copying of the piece of data directly from the first node to the second node.

Abstract: Provided are systems and methods for multi-layer caching of data. An example method commences with creating a data structure on top of an information retrieval library. The data structure is designed to access data associated with the information retrieval library through a local index. The method includes creating a plurality of ordered cache layers for the data accessed via the local index. The plurality of ordered cache layers are ordered based on frequency of access and a last layer is used to access the data over a network. The method then continues with receiving a search query. The method further includes determining that the search query is for the data associated with the last layer. The method further continues with executing, using the local index, the search query locally to retrieve a matching document from the data associated with the last layer remotely.