Abstract: A method for performing a backup in relation to a separated database, the method includes creating, by a processing circuit, a new backup log structured merge (LSM) tree of the separated database, wherein the new backup LSM tree is associated with a new point in time (PIT) and belongs to a group of LSM trees that comprise a primary LSM tree and one or more backup LSM trees that are associated with different PITs, and share a key value (KV) database that is mutable and is separated from the group; wherein the creating comprises storing the new backup LSM tree to a destination.

Abstract: The present teaching relates to a method, system and programming for recovering a data storage. In response to a failure of the data storage, a last checkpointed version is retrieved from a checkpoint file created previously for the failure of the data storage. A recovery table is updated to indicate the last checkpointed version and an incarnation number indicating a number of previous recovery of the data storage. The data storage is recovered based on the recovery table. A version number of the last checkpointed version is reset to zero upon a successful recovery of the data storage. The incarnation number is incremented.

Abstract: Methods, systems, and programming for retrieving content items for a search are described herein. In a non-limiting embodiment, a query including a plurality of terms may be received. For each of the plurality of terms, a posting list of one or more content items may be obtained. The posting list may include a ranked list of term scores corresponding to the one or more content items, each of the term scores being indicative of a level of relevance of a corresponding content item to a term associated with the posting list. A list of relevant content items for the query may be determined based on the term scores in each posting list for the one or more content items identified with respect to each term. At least one of the relevant content items may be provided as a response to the query.

Abstract: A method for updating a log structured merged (LSM) tree, the method includes (a) performing preemptive full merge operations at first LSM tree levels; and (b) performing capacity triggered merge operations at second LSM tree levels while imposing one or more restrictions; wherein the second LSM tree levels comprise a largest LSM tree level and one or more other second LSM tree levels that are larger from each first LSM tree level; wherein files of the one or more other second LSM tree levels are aligned with files of the largest LSM tree level.

Abstract: In one embodiment, a processor of a computing device receives a query. The computing device may compare a centroid of each of a plurality of clusters to the query such that a subset of the plurality of clusters is selected, each of the plurality of clusters having a set of data points. An assignment of the subset of the plurality of clusters may be communicated to a hardware accelerator of the computing device. A plurality of threads of the hardware accelerator of the computing device may generate one or more distance tables that store results of intermediate computations corresponding to the query and the subset of the plurality of clusters. The distance tables may be stored in shared memory of the hardware accelerator. A plurality of threads of the hardware accelerator may determine a plurality of data points using the distance tables. The processor may provide query results pertaining to at least a portion of the plurality of data points.

Abstract: Methods, systems, and programming for updating a search index are described herein. In a non-limiting embodiment, event data representing actions performed on a plurality of content items managed via the search index may be obtained. The event data may be filtered based on pre-determined criteria to generate updated event data, where the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index. An updated search index based on the updated event data may then be generated.

Abstract: The present teaching relates to a method, system and programming for operating a data storage. The data storage comprises of different portions including: a first portion having a plurality of metadata objects stored therein, each of the metadata objects being associated with a filter and corresponding to a range of keys, wherein at least one of the metadata objects is associated with a data structure, and a second portion having a plurality of files stored therein, each of the plurality of files being associated with one of the plurality of metadata objects; The data storage synchronizes a scan request with respect to one or more write requests based on a parameter associated with the scan request and each of the one or more write requests.

Abstract: Technologies for enhancing work scheduling of a big data framework. The technologies can include generating, in a database, configurable namespaces to be used by a work scheduling enhancement application to group together tasks of one or more big data clusters. The namespaces can be hierarchical. The technologies can also include linking in the database, by the application, related tasks with respective namespaces to categorize and group together the related tasks. The technologies can also include configuring, by the application, a display scheme for displaying error handling and root cause analysis of tasks of the one or more big data clusters. The technologies can also include generating or rendering, by the application, a GUI having a navigable hierarchal view for displaying the namespaces. The generation or rendering of the GUI can be based partially on the display scheme.

Abstract: The present teaching relates to a system and method for providing a response to a query related to a data sketch. A local data sketch is obtained in response to a synchronization parameter being set to a first value. The data sketch is updated based on the obtained local data sketch. The synchronization parameter is set to a second value that is different than the first value and thereafter propagated. A response to a query is generated based on the updated data sketch.

Abstract: The present teaching relates to a method and system generating a data sketch. A plurality of local data sketches are generated, wherein for each local data sketch, it is determined whether a propagation condition is satisfied. In response to the condition being satisfied, the local data sketch is processed when a synchronization parameter satisfies a criterion. The synchronization parameter is set to a predetermined value to enable propagation of the local data sketch to the data sketch. Upon the local data sketch being propagated to the data sketch, the above steps are repeated.

Abstract: Technologies for enhancing work scheduling of a big data framework. The technologies can include generating, in a database, configurable namespaces to be used by a work scheduling enhancement application to group together tasks of one or more big data clusters. The namespaces can be hierarchical. The technologies can also include linking in the database, by the application, related tasks with respective namespaces to categorize and group together the related tasks. The technologies can also include configuring, by the application, a display scheme for displaying error handling and root cause analysis of tasks of the one or more big data clusters. The technologies can also include generating or rendering, by the application, a GUI having a navigable hierarchal view for displaying the namespaces. The generation or rendering of the GUI can be based partially on the display scheme.

Abstract: Systems and methods for indexing content in a personal search system are disclosed. In one embodiment, the method comprises receiving a content item from a data source; extracting, from the content item, at least one document appearing in the content item, the at least one document including a first timestamp; storing the document in a content index; retrieving an event associated with the document from an event index, the event including a second timestamp and one or more attributes; generating an updated document based on the one or more attributes, the updated document including the second timestamp and the one or more attributes; and replacing the document stored in the content index with the updated document if the second timestamp is later than the first timestamp.

Abstract: In one embodiment, a processor of a computing device receives a query. The computing device may compare a centroid of each of a plurality of clusters to the query such that a subset of the plurality of clusters is selected, each of the plurality of clusters having a set of data points. An assignment of the subset of the plurality of clusters may be communicated to a hardware accelerator of the computing device. A plurality of threads of the hardware accelerator of the computing device may generate one or more distance tables that store results of intermediate computations corresponding to the query and the subset of the plurality of clusters. The distance tables may be stored in shared memory of the hardware accelerator. A plurality of threads of the hardware accelerator may determine a plurality of data points using the distance tables. The processor may provide query results pertaining to at least a portion of the plurality of data points.

Abstract: The present teaching relates to a method, system and programming for recovering a data storage. In response to a failure of the data storage, a last checkpointed version is retrieved from a checkpoint file created previously for the failure of the data storage. A recovery table is updated to indicate the last checkpointed version and an incarnation number indicating a number of previous recovery of the data storage. The data storage is recovered based on the recovery table. A version number of the last checkpointed version is reset to zero upon a successful recovery of the data storage. The incarnation number is incremented.

Abstract: Multi-thread systems and methods are described for concurrently handling requests to commit data updates to a database by a plurality of data transactions. The database preferably supports multi-versioning and the data transactions are preferably isolated by snapshot isolation. In one embodiment, concurrent and lock-free handling of requests to commit data updates includes performing two types of concurrent data conflict detection. A transaction proceeds to commit only if it passes both types of conflict detection. The first type of conflict detection is based on a hash map between data keys and their commit timestamps whereas the second type of conflict detection is based on a log that keeps track of the status of transactions whose requests to commit are actively being processed. In another embodiment, concurrent conflict detection for data items in concurrent transactions is broken down into buckets and locks are used for accessing each bucket.

Abstract: In one embodiment, a processor of a computing device receives a query. The computing device may compare a centroid of each of a plurality of clusters to the query such that a subset of the plurality of clusters is selected, each of the plurality of clusters having a set of data points. An assignment of the subset of the plurality of clusters may be communicated to a hardware accelerator of the computing device. A plurality of threads of the hardware accelerator of the computing device may generate one or more distance tables that store results of intermediate computations corresponding to the query and the subset of the plurality of clusters. The distance tables may be stored in shared memory of the hardware accelerator. A plurality of threads of the hardware accelerator may determine a plurality of data points using the distance tables. The processor may provide query results pertaining to at least a portion of the plurality of data points.

Abstract: The present teaching relates to a method, system and programming for searching a data storage. A key is extracted from a request and a metadata object associated with the key is identified. Further, a determination is made as to whether the metadata object is associated with a data structure stored in a first portion of the data storage. In response to a successful determination, the data structure is searched to retrieve a value associated with the key from the first portion. In response to an unsuccessful determination, a cache is searched to retrieve the value associated with the key, and in response to the key being absent in the cache, a file associated with the metadata object is searched to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.

Abstract: The present teaching relates to a method, system and programming for operating a data storage. The data storage comprises of different portions including: a first portion having a plurality of metadata objects stored therein, each of the metadata objects being associated with a filter and corresponding to a range of keys, wherein at least one of the metadata objects is associated with a data structure, and a second portion having a plurality of files stored therein, each of the plurality of files being associated with one of the plurality of metadata objects; The data storage synchronizes a scan request with respect to one or more write requests based on a parameter associated with the scan request and each of the one or more write requests.

Abstract: Methods, systems, and programming for updating a search index are described herein. In a non-limiting embodiment, event data representing actions performed on a plurality of content items managed via the search index may be obtained. The event data may be filtered based on pre-determined criteria to generate updated event data, where the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index. An updated search index based on the updated event data may then be generated.

Abstract: The present teaching relates to concurrent database operation. In one example, a plurality of requests which includes a scan request to obtain first data associated with a plurality of first keys stored in a database is received concurrently. A global version number is updated upon receipt of the scan request. The first data associated with the plurality of first keys is obtained based on the updated global version number. The first data is provided in response to the scan request.