Abstract: Techniques are described herein that are capable of performing a security action based on anomaly detection using AI model profiles and user profiles. AI model profiles (e.g., a model-session profile, a model-response profile, and/or a model-response profile) associated with AI model(s) are generated. User profiles (e.g., user-session profiles, user-prompt profiles, and/or user-response profiles) associated with users of the AI model(s) are generated. A security action is performed with regard to an incoming AI prompt as a result of a difference between the incoming AI prompt and one or more of the AI model profiles and/or one or more of the user profiles being greater than or equal to a difference threshold.

Abstract: This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.

Abstract: Techniques are described herein that are capable of triggering a security action based on an AI-generated recommendation of a code package. An AI model is caused to recommend an identified code package to resolve a coding problem by providing an AI prompt to the AI model. The AI prompt requests identification of a code package that is written in a programming language and that comprises a designated functionality that resolves the coding problem. A response to the AI prompt is received from the AI model. The response identifies the identified code package. Based at least on confirmation of non-existence of the identified code package or absence of publication of the identified code package in a verified code repository or a value of an attribute of the identified code package satisfying a criterion associated with non-trustworthiness, automatic execution of a security action with regard to the identified code package is triggered.

Abstract: This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.

Abstract: Systems and methods for detecting and mitigating prompt injection attacks on a generative LLM are disclosed. A deployment scenario is considered, in which the generative LLM supports a task automation function. Prompts are received and interpreted by the generative LLM, and outputs from the generative LLM are used to trigger automation actions. The prompts are constructed based on a combination of user input and external data and are, therefore, vulnerable to prompt injection attacks though manipulation of the external data. To mitigate this risk, a separate discriminative classification, decoupled from the generative LLM, engine is configured to identify malicious prompts, and filter out any malicious prompts before they reach the generative LLM.

Abstract: Described herein are technologies related to analyzing behavioral data of an entity in a cloud computing environment and determining suitability of providing the behavioral data to a computer-executable model that is configured to identify anomalous behavior of the entity. The technologies described herein improve performance of computer-executable models that are configured to detect anomalous behavior in a cloud computing environment.

Abstract: This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.