Abstract: A system, computer implemented method, and computer storage medium encoded with a computer program, for identifying newly trending topics in a data stream. An example method includes: receiving text documents forming part of a data stream from one or more servers; identifying terms within the received text documents; deriving from the identified terms, a set of terms identified as abnormal by virtue of having a relatively high frequency of occurrence within the text documents received in a recent period compared with that expected from their historic occurrence; creating a first set of one or more clusters, each cluster including a group of terms from the set of terms identified as abnormal which through their degree of co-occurrence in the received text documents are considered to relate to the same topic; and comparing clusters of a further set with the clusters of the first set to determine whether a cluster of the further set pertains to the same topic.

Abstract: A method, system, and computer program product for discriminating boilerplate text in documents, such as web pages. An example method includes: receiving documents structured as labelled text elements; generating a local language model for each labelled text element of the received documents; comparing local language models for different labelled text elements that have the same label; for each comparison of local language models, deriving a similarity indicator, and using the similarity indicators of all the comparisons to derive a similarity score for that label; using the similarity scores to determine labels associated with text elements comprising boilerplate text; and providing the textual content of the labelled text elements to a receiving computer system; and identifying the textual content of labelled text elements that include boilerplate text.

Abstract: A system, computer implemented method, and computer storage medium encoded with a computer program, for identifying newly trending topics in a data stream. An example method includes: receiving text documents forming part of a data stream from one or more servers; identifying terms within the received text documents; deriving from the identified terms, a set of terms identified as abnormal by virtue of having a relatively high frequency of occurrence within the text documents received in a recent period compared with that expected from their historic occurrence; creating a first set of one or more clusters, each cluster including a group of terms from the set of terms identified as abnormal which through their degree of co-occurrence in the received text documents are considered to relate to the same topic; and comparing clusters of a further set with the clusters of the first set to determine whether a cluster of the further set pertains to the same topic.

Abstract: A method, system, and computer program product for discriminating boilerplate text in documents, such as web pages. An example method includes: receiving documents structured as labelled text elements; generating a local language model for each labelled text element of the received documents; comparing local language models for different labelled text elements that have the same label; for each comparison of local language models, deriving a similarity indicator, and using the similarity indicators of all the comparisons to derive a similarity score for that label; using the similarity scores to determine labels associated with text elements comprising boilerplate text; and providing the textual content of the labelled text elements to a receiving computer system; and identifying the textual content of labelled text elements that include boilerplate text.