Abstract: Systems and methods are disclosed for managing a generative artificial intelligence (AI) model to improve performance. Managing the generative AI model includes using a second generative AI model to generate outputs from similar inputs and comparing the outputs of the generative AI models to determine their similarity. The second generative AI model is trained using fresher training data but may not be manually supervised and adjusted to the level of the generative AI model being managed. As such, an output of the second generative AI model is compared to an output of the managed generative AI model by a classification model to determine a relevance of the output from the managed generative AI model. An output of the classification model is used to perform various suitable policies to optimize the performance of the managed generative AI model, such as providing alternate outputs, preventing providing the output, or retraining the model.

Abstract: A method of score prediction uses hierarchical attention. Word features, positioning features, participant embedding features, and metadata are extracted from a transcript of a conversation. A word encoder vector is formed by multiplying weights of a word encoder layer to one or more word features. A sentence vector is formed by multiplying weights of a word attention layer to word encoder vectors. An utterance encoder vector is formed by multiplying weights of an utterance encoder layer to the sentence vector. A conversation vector is formed by multiplying weights of an utterance attention layer to utterance encoder vectors. The utterance encoder vector is combined with one or more positioning features and one or more participant embedding features. A predicted net promoter score is generated by multiplying weights of an output layer to the conversation vector combined with the metadata. The predicted net promoter score is presented in a list of conversations.

Abstract: Systems and methods for automatically assessing a quality of service for agents of a customer support system are disclosed. An example method may include retrieving historical conversations between the agents and users of the customer support system, receiving user comments for one or more of the historical conversations, identifying which of the received user comments includes keywords indicative of one or more quality of service attributes, generating transcripts of historical conversations associated with the identified user comments, training a machine learning model based at least in part on the generated transcripts and the user comments of the historical conversations associated with the identified user comments, providing a plurality of current conversations between agents and users of the customer support system to the trained machine learning model, and generating a behavioral score for each of the agents using the trained machine learning model.

Abstract: Systems and models are disclosed for determining a value over replacement feature (VORF) for one or more features of a machine learning model. An example method includes selecting one or more features used in the machine learning model, determining a comparison set of unused features not used in the machine learning model, for each unused feature in the comparison set, determining a difference in a specified metric when the selected one or more features are replaced by a corresponding unused feature from the comparison set, and determining the VORF to be the smallest difference in the specified metric.

Abstract: A computer-implemented method and system are provided to perform a machine learning pipeline process to produce an explainable machine learning model. A computing device may be configured to train a plurality of machine learning models with a set of respective feature datasets to generate an accuracy and explainability property for each trained model. The computing device may evaluate a plurality of the trained machine learning models and select a model as an explainable machine learning model based on at least one of the accuracy and the explainability property.

Abstract: The business segment associated with a merchant is automatically and accurately determined by applying machine learning techniques to actual financial documents associated with a merchant. In some examples, once the business segment associated with a merchant user of a data management system is identified, this information is used to identify potentially fraudulent and/or other criminal activity such as fraudulent merchants, criminal financial transactions, and fraudulent invoices.

Abstract: One or more embodiments includes storing a group email message in multiple email mailboxes to obtain multiple stored email messages, receiving, for a first stored email message in the stored email messages, an updated label from a first user computing device, detecting, in response to receiving the updated label, that the stored email messages matches the first stored email message, and storing the updated label with at least a subset of the stored email messages. The method further includes transmitting, with the updated label and to a second user computing system, a second stored email message in the stored email messages.