Abstract: A method may include: receiving a plurality of log-text examples from network devices and labels indicating if the log-text example is anomalous or not anomalous; identifying, from the log-text examples and labels, rules for weak annotation, and validation labels for a validation dataset and test labels for a test dataset; organizing log-text examples in the validation dataset into a plurality of time windows; concatenating log-text examples in each time window; providing the concatenated log-text examples and the rules for weak annotation to a weak annotation framework, to a LLM training dataset; training a LLM with the LLM training dataset; collecting runtime log-texts from the network devices for a period of time; concatenating the runtime log-texts for the period of time; and prompting the LLM for analysis with the concatenated log-texts, wherein the LLM outputs a response of anomaly or no anomaly and a confidence in the response.

Abstract: A method may include: receiving a plurality of log-text examples from network devices and labels indicating if the log-text example is anomalous or not anomalous; identifying, from the log-text examples and labels, rules for weak annotation, and validation labels for a validation dataset and test labels for a test dataset; organizing log-text examples in the validation dataset into a plurality of time windows; concatenating log-text examples in each time window; providing the concatenated log-text examples and the rules for weak annotation to a weak annotation framework, to a LLM training dataset; training a LLM with the LLM training dataset; collecting runtime log-texts from the network devices for a period of time; concatenating the runtime log-texts for the period of time; and prompting the LLM for analysis with the concatenated log-texts, wherein the LLM outputs a response of anomaly or no anomaly and a confidence in the response.

Abstract: Systems and methods for detection of hallucination in large language models are disclosed. According to an embodiment, a method may include: (1) receiving, by a computer program, a plurality of input texts, wherein each input text is a prompt for a large language model (LLM) and may include a slight perturbation from an initial input text; (2) generating, by the computer program and for each of the plurality of input texts, an input embedding vector; (3) providing, by the computer program, each input text to a large language model (LLM); (4) receiving, by the computer program and for each input text from the LLM, an output text; (5) generating, by the computer program and for each of the plurality of output texts, an output embedding vector; and (6) generating, by the computer program, a hallucination metric based on the input embedding vectors and the output embedding vectors.

Abstract: A method may include: receiving a set of retain samples comprising retain features to retain in a pretrained machine learning model, and a set of forget samples comprising forget features to remove from the pretrained machine learning model; providing the set of retain samples to the pretrained machine learning model resulting in a retain output and the set of forget samples to the pretrained machine learning model, resulting in a forget output; generating a set of retain weights and a set of forget weights based on the retain output and the forget output; freezing the set of retain weights; setting each forget weight to an initial state; executing a training epoch using the pretrained machine learning model and the retain samples that retrains the forget weights using the retain samples; combining the retrained forget weights with the retained weights to form an unlearned machine learning model.