Abstract: A method and a system for counteracting model drift in a Latent Dirichlet Allocation (LDA) model by maintaining freshness and accuracy in the LDA model by leveraging LDA classification and vector algebra to detect potential degradations are provided. The method entails determining whether an LDA model has drifted and therefore requires retraining based on measurements of cosine similarities of respective topics that correspond to two LDA models and a topic match entropy between the two LDA models that is determined based on the measurements.

Abstract: A method and a system for counteracting model drift in a Latent Dirichlet Allocation (LDA) model by maintaining freshness and accuracy in the LDA model by leveraging LDA classification and vector algebra to detect potential degradations are provided. The method includes: receiving a first corpus that includes a first set of textual data; generating a first set of topics by applying a first LDA model to the first corpus; generating, for each topic, a corresponding vector of words with associated weights; measuring a cosine similarity of each topic with respect to a second set of topics that is associated with a second LDA model; using the measured cosine similarities to determine a topic match entropy between the first LDA model and the second LDA model; and when the topic match entropy exceeds a predetermined threshold, determining that the first LDA model has drifted and requires retraining.