Abstract: Systems and methods for monitoring the quality of document reviews used in continuous active machine learning are described herein. Two orthogonal processes may be run simultaneously, asynchronously, and continuously. The first process performs continuous active machine learning for training machine classification models. The second process classifies documents that have been reviewed as part of the first process to generate classification scores of the reviewed documents. The original review may be compared to the classification scores using false negative and a false positive thresholds to identify documents that may have been incorrectly reviewed. A master review of identified documents is used to correct original reviews that were incorrect. Original incorrect reviews may be replaced in a training corpus by corrected reviews, and the models may be retrained using the corrected reviews.

Abstract: Systems and methods for improving search results from proprietary search engine technologies and proprietary machine classifiers, without ingesting, copying, or storing, the data to be searched, are described herein. A user sends a query to a proprietary search engines and gets a result document set back. The user may apply a user model for classifying a result document set to generate a result document for review of a user. The reviewed document may be added to a user training corpus, which is then used to retrain the user model. The retrained user model may be applied by the user to generate the next result document for user review and so on until the user model converges to generate relevant documents reliably. Once the user model converges, the user may apply the now reliable user model to generate multiple relevant documents for the user.

Abstract: Systems and methods for monitoring the quality of document reviews used in continuous active machine learning are described herein. Two orthogonal processes may be run simultaneously, asynchronously, and continuously. The first process performs continuous active machine learning for training machine classification models. The second process classifies documents that have been reviewed as part of the first process to generate classification scores of the reviewed documents. The original review may be compared to the classification scores using false negative and a false positive thresholds to identify documents that may have been incorrectly reviewed. A master review of identified documents is used to correct original reviews that were incorrect. Original incorrect reviews may be replaced in a training corpus by corrected reviews, and the models may be retrained using the corrected reviews.

Abstract: Systems and methods for improving search results from proprietary search engine technologies and proprietary machine classifiers, without ingesting, copying, or storing, the data to be searched, are described herein. A user sends a query to a proprietary search engines and gets a result document set back. The user may apply a user model for classifying a result document set to generate a result document for review of a user. The reviewed document may be added to a user training corpus, which is then used to retrain the user model. The retrained user model may be applied by the user to generate the next result document for user review and so on until the user model converges to generate relevant documents reliably. Once the user model converges, the user may apply the now reliable user model to generate multiple relevant documents for the user.

Abstract: Systems and methods for monitoring the quality of document reviews used in continuous active machine learning are described herein. Two orthogonal processes may be run simultaneously, asynchronously, and continuously. The first process performs continuous active machine learning for training machine classification models. The second process classifies documents that have been reviewed as part of the first process to generate classification scores of the reviewed documents. The original review may be compared to the classification scores using false negative and a false positive thresholds to identify documents that may have been incorrectly reviewed. A master review of identified documents is used to correct original reviews that were incorrect. Original incorrect reviews may be replaced in a training corpus by corrected reviews, and the models may be retrained using the corrected reviews.

Abstract: A method of forming a keyword based search query that uses a plurality of keywords, in which the keywords of the query are arranged into groups of purpose-related keywords, in which each keyword is associated with a designation of its relative importance. The keywords of a group may be identified manually by the searcher, or existing keywords in a group may be used to suggest or automatically add additional related keywords to the group. The keywords of a group need not be semantically related, but only related to a common purpose of the keyword group. Additional keywords can be suggested to the searcher, or automatically added to the groups. Suitable additional keywords may be identified by reference to previous searches in which the existing keywords were grouped with those additional keywords. Keywords may also be derived from text designated by the searcher, using all or portions of one or more documents or text blocks that the user identifies as describing a concept of interest to the searcher.