Abstract: Methods and systems include determining that a query is relevant to information that is unknown to a pre-trained language model. Outputs from adapter layers are added to outputs of respective transformer layers of the language model to infuse the language model with the information, such that the language model generates a response to the query that accounts for the information that is unknown to the pre-trained language model. An action is performed based on the response.

Abstract: Systems and methods are provided for adapting a pretrained language model to perform cybersecurity-specific named entity recognition and relation extraction. The method includes introducing a pretrained language model and a corpus of security text to a model adaptor, and generating a fine-tuned language model through unsupervised training utilizing the security text corpus. The method further includes combining a joint extraction model from a head for joint extraction with the fine-tuned language model to form an adapted joint extraction model that can perform entity and relation label prediction. The method further includes applying distant labels to security text in the corpus of security text to produce security text with distant labels, and performing Distant Supervision Training for joint extraction on the adapted joint extraction model using the security text to transform the adapted joint extraction model into a Security Language Model for name-entity recognition (NER) and relation extraction (RE).

Abstract: Methods and systems for prompting a Large Language Model (LLM) with a set of text data outside pre-inference trained categories and a test prompt for an initial parameter which has a known ground truth, calculating an uncertainty of an LLM's output, selecting another LLM model parameter and calculating the total uncertainty of the LLM's output with the other LLM model parameter. The methods and systems further include prompting the LLM with another test prompt, with the initial LLM parameter and the other LLM parameter, and calculating the total uncertainty of the LLM's output for initial LLM model parameter and the other LLM model parameter, decomposing the total uncertainty of the LLM into Aleatoric Uncertainty (AU) and Epistemic Uncertainty (EU) components, and rating the total uncertainty of the LLM, using the decomposed total uncertainty as a metric.

Abstract: Methods and systems for root cause analysis include combining system logs and system metrics into time-series data. Individual root cause analysis is performed to determine individual causal scores for respective system entities. Topological root cause analysis is performed to capture topological patterns of system anomalies. The individual causal scores and the topological patterns are integrated by a weighted sum. A corrective action is performed on an entity identified based on the weighted sum.

Abstract: A method for implementing a self-attentive encoder-decoder transformer framework for anomaly detection in event sequences is presented. The method includes feeding event content information into a content-awareness layer to generate event representations, inputting, into an encoder, event sequences of two hierarchies to capture long-term and short-term patterns and to generate feature maps, adding, in the decoder, a special sequence token at a beginning of an input sequence under detection, during a training stage, applying a one-class objective to bound the decoded special sequence token with a reconstruction loss for sequence forecasting using the generated feature maps from the encoder, and during a testing stage, labeling any event representation whose decoded special sequence token lies outside a hypersphere as an anomaly.

Abstract: Methods and systems for model compression include determining importance values for respective parameters in a pre-trained model corresponding to general knowledge of the pre-trained model. Loss values are determined for removal of the parameters based on the importance values and a regularization term corresponding to domain-specific knowledge. Parameters are pruned from the pre-trained model based on the loss values to create a pruned model.

Abstract: Systems and methods for training a time-series-language (TSLa) model adapted for domain-specific tasks. An encoder-decoder neural network can be trained to tokenize time-series data to obtain a discrete-to-language embedding space. The TSLa model can learn a linear mapping function by concatenating token embeddings from the discrete-to-language embedding space with positional encoding to obtain mixed-modality token sequences. Token augmentation can transform the tokens from the mixed-modality token sequences with to obtain augmented tokens. The augmented tokens can train the TSLa model using a computed token likelihood to predict next tokens for the mixed-modality token sequences to obtain a trained TSLa model. A domain-specific dataset can fine-tune the trained TSLa model to adapt the trained TSLa model to perform a domain-specific task.

Abstract: Methods and systems include annotating a set of training data to indicate tokens that are sensitive. Instructions are generated based on the training data, including original token sequences and respective substituted token sequences. A language model is fine-tuned using the instructions with a penalty-based loss function to generate a privacy-protected language model.

Abstract: Systems and methods for log representation learning for automated system maintenance. An optimized parser can transform collected system logs into log templates. A tokenizer can tokenize the log templates partitioned into time windows to obtain log template tokens. The log template tokens can train a language model (LM) with deep learning to obtain a trained LM. The trained LM can detect anomalies from system logs to obtain detected anomalies. A corrective action can be performed on a monitored entity based on the detected anomalies.

Abstract: Systems and methods for a demonstration uncertainty-based artificial intelligence model for open information extraction. A large language model (LLM) can generate initial structured sentences using an initial prompt for a domain-specific instruction extracted from an unstructured text input. Structural similarities between the initial structured sentences and sentences from a training dataset can be determined to obtain structurally similar sentences. The LLM can identify relational triplets from combinations of tokens from generated sentences using and the structurally similar sentences. The relational triplets can be filtered based on a calculated demonstration uncertainty to obtain a filtered triplet list. A domain-specific task can be performed using the filtered triplet list to assist the decision-making process of a decision-making entity.

Abstract: Systems and methods for optimizing large language models (LLM) with domain-oriented model compression. Importance weights for general knowledge in a trained LLM, pretrained with deep learning, can be determined by computing the error when removing a weight from the trained LLM. The trained LLM can be iteratively optimized to obtain a domain-compressed LLM with domain knowledge while maintaining general knowledge by: fine-tuning the trained LLM iteratively with domain knowledge using the importance weights for general knowledge to obtain a fine-tuned LLM; determining importance weights for domain knowledge in the LLM with a regularization term by using gradient descent to optimize parameters when the fine-tuned LLM is trained with domain knowledge; and pruning learned knowledge based on importance weights for domain knowledge. A corrective action can be performed on a monitored entity using the domain-compressed LLM.

Abstract: A method for detecting business intent from a business intent corpus by employing an Intent Detection via Multi-hop Unified Syntactic Graph (IDMG) is presented. The method includes parsing each text sample representing a business need description to extract syntactic information including at least tokens and words, tokenizing the words of the syntactic information to generate sub-words for each of the words by employing a multi-lingual pre-trained language model, aligning the generated sub-words to the tokens of the syntactic information to match ground-truth intent actions and objects to the tokenized sub-words, generating a unified syntactic graph, encoding, via a multi-hop unified syntactic graph encoder, the unified syntactic graph to generate an output, and predicting an intent action and object from the output.

Abstract: Systems and methods are provided for extracting relations from text data, including collecting labeled text data from diverse sources, including digital archives and online repositories, each source including sentences annotated with detailed grammatical structures. Initial relational data is generated from the grammatical structures by applying advanced parsing and machine learning techniques using a sophisticated rule-based algorithm. Training sets are generated for enhancing the diversity and complexity of a relation dataset by applying data augmentation techniques to the initial relational data. A neural network model is trained using an array of semantically equivalent but syntactically varied prompt templates designed to test and refine linguistic capabilities of a model. A final relation extraction output is determined by implementing a vote-based decision system integrating statistical analysis and utilizing a weighted voting mechanism to optimize extraction accuracy and reliability.

Abstract: Methods and systems for information extraction include configuring a language model with an information extraction instruction prompt and at least one labeled example prompt. Configuration of the language model is validated using at least one validation prompt. Errors made by the language model in response to the at least one validation prompt are corrected using a correction prompt. Information extraction is performed on an unlabeled sentence using the language model to identify a relation from the unlabeled sentence. An action is performed responsive to the identified relation.

Abstract: Systems and methods are provided for detecting and resolving non-synchronization in a complex system, including acquiring monitoring data from multiple computers and devices within the complex system, preparing the acquired data by aligning data sequences from different sources based on timestamps, segmenting the prepared data into time windows, and extracting a plurality of features from the data within each of the time windows. Significant features are selected from the extracted features based on their relevance to non-synchronization detection and detection algorithms are applied to the selected features to identify non-synchronization events within the system. Alerts are generated, responsive to the detection of non-synchronization events, which trigger targeted, automatic corrective measures including adjusting particular system parameters to resolve the non-synchronization events and prevent occurrence of future non-synchronization events for enhanced stability and performance of the complex system.

Abstract: Systems and methods are provided for dynamic prompt tuning in image processing, including decomposing a received image into segments sized to balance detail retention and computational efficiency for processing by an embedding algorithm designed for token generation, generating tokenized image data by transforming each of the decomposed segments into a sequence of tokens using an embedding process that includes a convolutional neural network, and dynamically computing parameters for inserting prompts into the sequence of tokens, including a position and length of the prompts, utilizing a one-layer neural network combined with a continuous relaxation of a discrete distribution for optimizing categorical decision-making. Soft prompts are created based on the dynamically computed parameters and the soft prompts are integrated with the tokenized image data. The integrated image data and prompts are processed using a pretrained vision model with a frozen backbone to enhance image feature recognition.

Abstract: Systems and methods are provided for Cross-lingual Transfer Interpretation (CTI). The method includes receiving text corpus data including premise-hypothesis pairs with a relationship label in a source language, and conducting a source to target language translation. The method further includes performing a feature importance extraction, where an integrated gradient is applied to assign an importance score to each input feature, and performing a cross-lingual feature alignment, where tokens in the source language are aligned with tokens in the target language for both the premise and the hypothesis based on semantic similarity. The method further includes performing a qualitative analysis, where the importance score of each token can be compared between the source language and the target language according to a feature alignment result.

Abstract: A computer-implemented method is provided for cross-lingual transfer. The method includes randomly masking a source corpus and a target corpus to obtain a masked source corpus and a masked target corpus. The method further includes tokenizing, by pretrained Natural Language Processing (NLP) models, the masked source corpus and the masked target corpus to obtain source tokens and target tokens. The method also includes transforming the source tokens and the target tokens into a source dependency parsing tree and a target dependency parsing tree. The method additionally includes inputting the source dependency parsing tree and the target dependency parsing tree into a graph encoder pretrained on a translation language modeling task to extract common language information for transfer. The method further includes fine-tuning the graph encoder and a down-stream network for a specific NLP down-stream task.

Abstract: Methods and systems for natural language processing include generating an encoder that includes a global part and a local part, where the global part encodes multi-hop relations between words in an input and where the local part encodes one-hop relations between words in the input. The encoder is trained to form a graph that represents tokens of an input text as nodes and that represents relations between the tokens as edges between the nodes.

Abstract: Methods and systems for anomaly detection include encoding a multivariate time series and a multi-type event sequence using respective transformers and an aggregation network to generate a feature vector. Anomaly detection is performed using the feature vector to identify an anomaly within a system. A corrective action is performed responsive to the anomaly to correct or mitigate an effect of the anomaly. The detected anomaly can be used in a healthcare context to support decision making by medical professionals with respect to the treatment of a patient. The encoding may include machine learning models to implement the transformers and the aggregation network using deep learning.