Abstract: The present teaching relates to a method, system and programming for operating a data storage. The data storage comprises of different portions including: a first portion having a plurality of metadata objects stored therein, each of the metadata objects being associated with a filter and corresponding to a range of keys, wherein at least one of the metadata objects is associated with a data structure, and a second portion having a plurality of files stored therein, each of the plurality of files being associated with one of the plurality of metadata objects; The data storage synchronizes a scan request with respect to one or more write requests based on a parameter associated with the scan request and each of the one or more write requests.

Abstract: In one embodiment, a search engine may generate and store a plurality of search index segments such that each of the search index segments is stored in a corresponding one of a plurality of heaps of memory. The plurality of search index segments may include inverted index segments mapping content to documents containing the content. A garbage collection module may release one or more heaps of the memory.

Abstract: Methods, systems, and programming for updating a search index are described herein. In a non-limiting embodiment, event data representing actions performed on a plurality of content items managed via the search index may be obtained. The event data may be filtered based on pre-determined criteria to generate updated event data, where the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index. An updated search index based on the updated event data may then be generated.

Abstract: Briefly, embodiments disclosed herein relate to managing transactions in a distributed content transaction system.

Abstract: Systems and methods for indexing content in a personal search system are disclosed. In one embodiment, the method comprises receiving a content item from a data source; extracting, from the content item, at least one document appearing in the content item, the at least one document including a first timestamp; storing the document in a content index; retrieving an event associated with the document from an event index, the event including a second timestamp and one or more attributes; generating an updated document based on the one or more attributes, the updated document including the second timestamp and the one or more attributes; and replacing the document stored in the content index with the updated document if the second timestamp is later than the first timestamp.

Abstract: Methods, systems, and programming for accessing data are described herein. In a non-limiting embodiment, a request associated with a key may be received. A memory chunk in a first memory site associated with the key may be located. An entry in the memory chunk corresponding to the key may be determined. A first pointer to the key corresponding to a first location of the key within a buffer at a second memory site may be obtained from the entry. The key stored at the first location may be retrieved from the buffer at the second memory site.

Abstract: The present teaching relates to a method, system and programming for operating a data storage. The data storage comprises of different portions including: a first portion having a plurality of metadata objects stored therein, each of the metadata objects being associated with a filter and corresponding to a range of keys, wherein at least one of the metadata objects is associated with a data structure, and a second portion having a plurality of files stored therein, each of the plurality of files being associated with one of the plurality of metadata objects; The data storage synchronizes a scan request with respect to one or more write requests based on a parameter associated with the scan request and each of the one or more write requests.

Abstract: The present teaching relates to a method, system and programming for searching a data storage. A key is extracted from a request and a metadata object associated with the key is identified. Further, a determination is made as to whether the metadata object is associated with a data structure stored in a first portion of the data storage. In response to a successful determination, the data structure is searched to retrieve a value associated with the key from the first portion. In response to an unsuccessful determination, a cache is searched to retrieve the value associated with the key, and in response to the key being absent in the cache, a file associated with the metadata object is searched to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.

Abstract: Systems and methods for indexing content in a personal search system are disclosed. In one embodiment, the method comprises receiving a content item from a data source; extracting, from the content item, at least one document appearing in the content item, the at least one document including a first timestamp; storing the document in a content index; retrieving an event associated with the document from an event index, the event including a second timestamp and one or more attributes; generating an updated document based on the one or more attributes, the updated document including the second timestamp and the one or more attributes; and replacing the document stored in the content index with the updated document if the second timestamp is later than the first timestamp.

Abstract: Technologies for enhancing work scheduling of a big data framework. The technologies can include generating, in a database, configurable namespaces to be used by a work scheduling enhancement application to group together tasks of one or more big data clusters. The namespaces can be hierarchical. The technologies can also include linking in the database, by the application, related tasks with respective namespaces to categorize and group together the related tasks. The technologies can also include configuring, by the application, a display scheme for displaying error handling and root cause analysis of tasks of the one or more big data clusters. The technologies can also include generating or rendering, by the application, a GUI having a navigable hierarchal view for displaying the namespaces. The generation or rendering of the GUI can be based partially on the display scheme.

Abstract: The present teaching relates to a system and method for providing a response to a query related to a data sketch. A local data sketch is obtained in response to a synchronization parameter being set to a first value. The data sketch is updated based on the obtained local data sketch. The synchronization parameter is set to a second value that is different than the first value and thereafter propagated. A response to a query is generated based on the updated data sketch.

Abstract: The present teaching relates to a method and system generating a data sketch. A plurality of local data sketches are generated, wherein for each local data sketch, it is determined whether a propagation condition is satisfied. In response to the condition being satisfied, the local data sketch is processed when a synchronization parameter satisfies a criterion. The synchronization parameter is set to a predetermined value to enable propagation of the local data sketch to the data sketch. Upon the local data sketch being propagated to the data sketch, the above steps are repeated.

Abstract: In one embodiment, a processor of a computing device receives a query. The computing device may compare a centroid of each of a plurality of clusters to the query such that a subset of the plurality of clusters is selected, each of the plurality of clusters having a set of data points. An assignment of the subset of the plurality of clusters may be communicated to a hardware accelerator of the computing device. A plurality of threads of the hardware accelerator of the computing device may generate one or more distance tables that store results of intermediate computations corresponding to the query and the subset of the plurality of clusters. The distance tables may be stored in shared memory of the hardware accelerator. A plurality of threads of the hardware accelerator may determine a plurality of data points using the distance tables. The processor may provide query results pertaining to at least a portion of the plurality of data points.

Abstract: The present teaching relates to committing transactions over a database in a semi-distributed manner. In one example, a transaction is executed to access a plurality of keys stored in the database. A request is transmitted to a transaction management engine to commit the transaction. The transaction management engine performs a conflict check between the transaction and one or more transactions running concurrently with the transaction, and generates an instruction based on the conflict check. The instruction is received from the transaction management engine. Based on the instruction, one or more of the plurality of keys are updated in the database or data added by the transaction to the database are erased.

Abstract: In one embodiment, a processor of a computing device receives a query. The computing device may compare a centroid of each of a plurality of clusters to the query such that a subset of the plurality of clusters is selected, each of the plurality of clusters having a set of data points. An assignment of the subset of the plurality of clusters may be communicated to a hardware accelerator of the computing device. A plurality of threads of the hardware accelerator of the computing device may generate one or more distance tables that store results of intermediate computations corresponding to the query and the subset of the plurality of clusters. The distance tables may be stored in shared memory of the hardware accelerator. A plurality of threads of the hardware accelerator may determine a plurality of data points using the distance tables. The processor may provide query results pertaining to at least a portion of the plurality of data points.

Abstract: Methods, systems, and programming for updating a search index are described herein. In a non-limiting embodiment, event data representing actions performed on a plurality of content items managed via the search index may be obtained. The event data may be filtered based on pre-determined criteria to generate updated event data, where the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index. An updated search index based on the updated event data may then be generated.

Abstract: Methods, systems, and programming for accessing data are described herein. In a non-limiting embodiment, a request associated with a key may be received. A memory chunk in a first memory site associated with the key may be located. An entry in the memory chunk corresponding to the key may be determined. A first pointer to the key corresponding to a first location of the key within a buffer at a second memory site may be obtained from the entry. The key stored at the first location may be retrieved from the buffer at the second memory site.

Abstract: Methods and apparatus for performing top-k query processing include pruning a list of documents to identify a subset of the list of documents, where pruning includes, for other query terms in the set of query terms, skipping a document in the list of documents based, at least in part, on the contribution of the query term to the score of the corresponding document and the term upper bound for each other query term, in the set of query terms, that matches the document.

Abstract: Methods, systems, and programming for retrieving content items for a search are described herein. In a non-limiting embodiment, a query including a plurality of terms may be received. For each of the plurality of terms, a posting list of one or more content items may be obtained. The posting list may include a ranked list of term scores corresponding to the one or more content items, each of the term scores being indicative of a level of relevance of a corresponding content item to a term associated with the posting list. A list of relevant content items for the query may be determined based on the term scores in each posting list for the one or more content items identified with respect to each term. At least one of the relevant content items may be provided as a response to the query.

Abstract: Systems and methods for indexing content in a personal search system are disclosed. In one embodiment, the method comprises receiving a content item from a data source; extracting, from the content item, at least one document appearing in the content item, the at least one document including a first timestamp; storing the document in a content index; retrieving an event associated with the document from an event index, the event including a second timestamp and one or more attributes; generating an updated document based on the one or more attributes, the updated document including the second timestamp and the one or more attributes; and replacing the document stored in the content index with the updated document if the second timestamp is later than the first timestamp.