Abstract: Systems and methods for aligning a language model (LM) are disclosed herein. An example method is performed by one or more processors of a computing system. The example method may include: receiving, over a communications network coupled to the computing system, an LM including a set of neural network parameters; obtaining a set of delta values representative of a difference between a prior LM's neural network parameters before a performance of an alignment operation and the prior LM's neural network parameters after the performance of the alignment operation, the alignment operation performed using alignment data for aligning the prior LM's output with at least one of a tone, voice, or safety preference; and adjusting the LM's neural network parameters based on the set of delta values such that, without undergoing the alignment operation, an expected output of the LM aligns with the at least one tone, voice, or safety preference.

Abstract: Certain aspects of the disclosure provide techniques for training data generation for large language model (LLM) training and/or benchmarking. A method generally includes obtaining domain data associated with a domain; generating a prompt based on configuration parameter(s) included in a configuration file, the prompt comprising: a request to generate training data for the domain based on the domain data, wherein the training data comprises: a first plurality of question and answer pairs; a conversation comprising a first plurality of questions and a first plurality of answers corresponding to the first plurality of questions; a second plurality of questions; or a question and a second plurality of answers corresponding to the question; guideline(s) for generating the training data; example training data for the domain; and the domain data; prompting an LLM with the prompt to generate the training data; and receiving, from the LLM, the training data based on the prompt.

Abstract: A method classifies feedback from transcripts. The method includes receiving an utterance from a transcript from a communication session and processing the utterance with a classifier model to identify a topic label for the utterance. The classifier model is trained to identify topic labels for training utterances. The topic labels correspond to topics of clusters of the training utterances. The training utterances are selected using attention values for the training utterances and clustered using encoder values for the utterances. The method further includes routing the communication session using the topic label for the utterance.

Abstract: A method classifies feedback from transcripts. The method includes receiving an utterance from a transcript from a communication session and processing the utterance with a classifier model to identify a topic label for the utterance. The classifier model is trained to identify topic labels for training utterances. The topic labels correspond to topics of clusters of the training utterances. The training utterances are selected using attention values for the training utterances and clustered using encoder values for the utterances. The method further includes routing the communication session using the topic label for the utterance.

Abstract: Embodiments disclosed herein may extract trending topics from phone call transcripts or any type of text data. The phone call transcripts may be collected for a time period and the time period may be divided into time spans. For each time span having more than a threshold number of phone call transcripts, n-grams from the phone call transcripts may be extracted. The extracted n-grams may be contextually clustered by converting the n-grams into their embedding vectors, reducing the dimensionality of the embedding vectors, and clustering similar reduced dimensionality embedding vectors. Normalized occurrences of one or more clusters may be generated. The recent mean of the number of occurrences of the normalized clusters may be compared with the historical mean and offset by historical standard deviation to generate a modified Z-score. N-grams corresponding to the clusters with high Z-scores may be identified as trending topics.

Abstract: A method of score prediction uses hierarchical attention. Word features, positioning features, participant embedding features, and metadata are extracted from a transcript of a conversation. A word encoder vector is formed by multiplying weights of a word encoder layer to one or more word features. A sentence vector is formed by multiplying weights of a word attention layer to word encoder vectors. An utterance encoder vector is formed by multiplying weights of an utterance encoder layer to the sentence vector. A conversation vector is formed by multiplying weights of an utterance attention layer to utterance encoder vectors. The utterance encoder vector is combined with one or more positioning features and one or more participant embedding features. A predicted net promoter score is generated by multiplying weights of an output layer to the conversation vector combined with the metadata. The predicted net promoter score is presented in a list of conversations.

Abstract: Systems and methods are disclosed for managing a generative artificial intelligence (AI) model to improve performance. Managing the generative AI model includes using a second generative AI model to generate outputs from similar inputs and comparing the outputs of the generative AI models to determine their similarity. The second generative AI model is trained using fresher training data but may not be manually supervised and adjusted to the level of the generative AI model being managed. As such, an output of the second generative AI model is compared to an output of the managed generative AI model by a classification model to determine a relevance of the output from the managed generative AI model. An output of the classification model is used to perform various suitable policies to optimize the performance of the managed generative AI model, such as providing alternate outputs, preventing providing the output, or retraining the model.

Abstract: Embodiments disclosed herein may extract trending topics from phone call transcripts or any type of text data. The phone call transcripts may be collected for a time period and the time period may be divided into time spans. For each time span having more than a threshold number of phone call transcripts, n-grams from the phone call transcripts may be extracted. The extracted n-grams may be contextually clustered by converting the n-grams into their embedding vectors, reducing the dimensionality of the embedding vectors, and clustering similar reduced dimensionality embedding vectors. Normalized occurrences of one or more clusters may be generated. The recent mean of the number of occurrences of the normalized clusters may be compared with the historical mean and offset by historical standard deviation to generate a modified Z-score. N-grams corresponding to the clusters with high Z-scores may be identified as trending topics.

Abstract: A method of score prediction uses hierarchical attention. Word features, positioning features, participant embedding features, and metadata are extracted from a transcript of a conversation. A word encoder vector is formed by multiplying weights of a word encoder layer to one or more word features. A sentence vector is formed by multiplying weights of a word attention layer to word encoder vectors. An utterance encoder vector is formed by multiplying weights of an utterance encoder layer to the sentence vector. A conversation vector is formed by multiplying weights of an utterance attention layer to utterance encoder vectors. The utterance encoder vector is combined with one or more positioning features and one or more participant embedding features. A predicted net promoter score is generated by multiplying weights of an output layer to the conversation vector combined with the metadata. The predicted net promoter score is presented in a list of conversations.