Abstract: Interviewing, for generation of training data from skilled personnel, is automated in various ways. The training data is intended to train other ML tools to emulate workflow elicited from the skilled personnel. A trainee machine-learning (ML) tool can be provided representations of interviews and trained to perform a prediction function. The trained tool can be deployed to participate in conducting additional interviews of skilled personnel. The deployed tool can act as an independent interviewer or as a partner to other interviewers or evaluators, and can give feedback to other interviewers or receive feedback from evaluators. Interview representations can be annotated, e.g. with workflow maps, topic allocations, or commentary. Training of interviewers, evaluators, and annotators is disclosed, as also synthesis of interview representations.

Abstract: Recorded sessions of skilled personnel at work are used to train a first machine learning (ML) tool to extract workflow maps. These maps are used to train a second ML tool to emulate at least one workflow. The first ML tool is trained to predict an annotator's output. Either ML tool can be a copilot having a microservice network architecture. Further, complex sets of workflows can be subdivided for efficient support with small ML tools for (1) workflow map extraction and (2) emulation. Based on segment demarcation accompanying a recording, segments are assigned to specialized ML extraction tools for workflow map extraction. Extracted workflow maps are used to train workflow emulator(s). Similarly, specialized ML emulation tools can emulate respective tasks. A distribution microservice can identify a task to be emulated and can invoke the appropriate specialized ML tool. Similar microservice network architectures support both applications.

Abstract: Recorded sessions of skilled personnel at work are used to train a first machine learning (ML) tool to extract workflow maps. These maps are used to train a second ML tool to emulate at least one workflow. The first ML tool is trained to predict an annotator's output. Either ML tool can be a copilot having a microservice network architecture. Further, complex sets of workflows can be subdivided for efficient support with small ML tools for (1) workflow map extraction and (2) emulation. Based on segment demarcation accompanying a recording, segments are assigned to specialized ML extraction tools for workflow map extraction. Extracted workflow maps are used to train workflow emulator(s). Similarly, specialized ML emulation tools can emulate respective tasks. A distribution microservice can identify a task to be emulated and can invoke the appropriate specialized ML tool. Similar microservice network architectures support both applications.

Abstract: Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. The microservice network architecture supports flexible, customizable, or dynamically determinable dataflow from client input to corresponding output. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while significantly reducing computation time and hardware requirements, even to a single compute node with a single GPU. Examples incorporate a qualification microservice to test data, destined for a downstream microservice, for conformance with the copilot's competency. A knowledge graph of a corpus of documents is built, visualized, and pruned. The data is tested for conformance with the pruned graph representation, and non-conforming data is excluded from the dataflow. Variations and additional techniques are disclosed.

Abstract: Apparatus and methods are disclosed for retrieving data, responsive to received input, at a microservice supporting an application programming interface (API). A score is generated, indicating likeness of semantic content, between each of multiple candidate API-conforming queries and the received input. Queries are selected based on their scores, and executed on a live repository. Based on retrieved data, a response to the input is formulated and transmitted. Disclosed techniques are suitable for a data producer front end in a copilot having a microservice network architecture. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while computation time and hardware requirements are significantly reduced, even to a single compute node with a single GPU. One or more data producers can provide a retrieval microservice with access to various databases having respective APIs, to extend the copilot's reach. Variations and additional techniques are disclosed.

Abstract: Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. The microservice network architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while significantly reducing hardware requirements and computation time. Disclosed examples incorporate microservices for expansion, retrieval, embedding, and evaluation, in addition to one or more core microservices. Optionally, intermodal I/O, multiple data repositories, competency qualification, or human feedback can be supported. Multiple core microservices can support varying client authorizations or cognitive functions. The disclosed architecture supports any major LLM use case and can be deployed on a single compute node with a single GPU.

Abstract: Methods and apparatus are disclosed for providing security for a target machine-learning (ML) tool and its host system. Input data is fed in parallel to a second ML tool. Fingerprints of the second ML tool are used to monitor changes in the second tool. Fingerprint changes above a threshold indicate anomalous input data and warn of possible threat to the target tool. Anomaly detection enables diagnosis and remediation. Compact fingerprints are easy to handle, and hide details of the underlying tool. Concurrently, fingerprints are large enough to be sensitive to localized variations within the tool. Alternative embodiments monitor fingerprints of the target tool itself. Further embodiments monitor input or output data streams for sensitive data using a trained ML classifier. Variations and applications are disclosed.

Abstract: Methods and apparatus are disclosed for providing security for a target machine-learning (ML) tool and its host system. Input data is fed in parallel to a second ML tool. Fingerprints of the second ML tool are used to monitor changes in the second tool. Fingerprint changes above a threshold indicate anomalous input data and warn of possible threat to the target tool. Anomaly detection enables diagnosis and remediation. Compact fingerprints are easy to handle, and hide details of the underlying tool. Concurrently, fingerprints are large enough to be sensitive to localized variations within the tool. Alternative embodiments monitor fingerprints of the target tool itself. Further embodiments monitor input or output data streams for sensitive data using a trained ML classifier. Variations and applications are disclosed.

Abstract: Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. This architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved, while significantly reducing computation time and hardware requirements, even to a single compute node with a single GPU. Examples incorporate a retrieval microservice, as least one data producer, and a core microservice. Based on client input, the retrieval microservice can perform multiple iterations of retrieval augmented generation (RAG). At each iteration, output (based on any preceding iterations' results or the client input) is transmitted to a data producer, and results received therefrom. Eventually, based on these results, an output is transmitted toward the core microservice for generation of a response to the client input.

Abstract: Methods and apparatus are disclosed for customizing a copilot. Document records related to copilot objectives or tasks are obtained and used to identify corresponding data sources. Data sources can be integrated into data producers or data repositories, to be used by a retrieval microservice. Data producers and other microservices are individually fine-tuned for the custom application, before or after integration into the copilot. The integrated copilot is tested end-to-end, and can be further refined. Disclosed techniques range from fully automated to human-in-the-loop (e.g. guided by an expert) to fully interactive. Some techniques produce tools which can automate certain customization operations. Training data, tasks, or document records blend expert-generated, developer-generated, or synthesized items.

Abstract: Methods and apparatus are disclosed for customizing a copilot or other machine learning tool. Document records related to copilot objectives or tasks are obtained and used to identify corresponding data sources. Data sources can be integrated into data producers or data repositories, to be used by a retrieval microservice. Data producers and other microservices are individually fine-tuned for the custom application, before or after integration into the copilot. The integrated copilot is tested end-to-end, and can be further refined. Disclosed techniques range from fully automated to human-in-the-loop (e.g. guided by an expert) to fully interactive. Some techniques produce tools which can automate certain customization operations. Training data, tasks, or document records blend expert-generated, developer-generated, or synthesized items.

Abstract: Methods and apparatus are disclosed for customizing a copilot or other machine learning tool. Document records related to copilot objectives or tasks are obtained and used to identify corresponding data sources. Data sources can be integrated into data producers or data repositories, to be used by a retrieval microservice. Data producers and other microservices are individually fine-tuned for the custom application, before or after integration into the copilot. The integrated copilot is tested end-to-end, and can be further refined. Disclosed techniques range from fully automated to human-in-the-loop (e.g. guided by an expert) to fully interactive. Some techniques produce tools which can automate certain customization operations. Training data, tasks, or document records blend expert-generated, developer-generated, or synthesized items.

Abstract: Interviewing, for generation of training data from skilled personnel, is automated in various ways. The training data is intended to train other ML tools to emulate workflow elicited from the skilled personnel. A trainee machine-learning (ML) tool can be provided representations of interviews and trained to perform a prediction function. The trained tool can be deployed to participate in conducting additional interviews of skilled personnel. The deployed tool can act as an independent interviewer or as a partner to other interviewers or evaluators, and can give feedback to other interviewers or receive feedback from evaluators. Interview representations can be annotated, e.g. with workflow maps, topic allocations, or commentary. Training of interviewers, evaluators, and annotators is disclosed, as also synthesis of interview representations.

Abstract: Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. The microservice network architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while significantly reducing hardware requirements and computation time. Disclosed examples incorporate microservices for expansion, retrieval, embedding, and evaluation, in addition to one or more core microservices. Optionally, intermodal I/O, multiple data repositories, competency qualification, or human feedback can be supported. Multiple core microservices can support varying client authorizations or cognitive functions. The disclosed architecture supports any major LLM use case and can be deployed on a single compute node with a single GPU.

Abstract: Recorded sessions of skilled personnel at work are used to train a first machine learning (ML) tool to extract workflow maps. These maps are used to train a second ML tool to emulate at least one workflow. The first ML tool is trained to predict an annotator's output. Either ML tool can be a copilot having a microservice network architecture. Further, complex sets of workflows can be subdivided for efficient support with small ML tools for (1) workflow map extraction and (2) emulation. Based on segment demarcation accompanying a recording, segments are assigned to specialized ML extraction tools for workflow map extraction. Extracted workflow maps are used to train workflow emulator(s). Similarly, specialized ML emulation tools can emulate respective tasks. A distribution microservice can identify a task to be emulated and can invoke the appropriate specialized ML tool. Similar microservice network architectures support both applications.

Abstract: Methods and apparatus are disclosed for customizing a copilot. Document records related to copilot objectives or tasks are obtained and used to identify corresponding data sources. Data sources can be integrated into data producers or data repositories, to be used by a retrieval microservice. Data producers and other microservices are individually fine-tuned for the custom application, before or after integration into the copilot. The integrated copilot is tested end-to-end, and can be further refined. Disclosed techniques range from fully automated to human-in-the-loop (e.g. guided by an expert) to fully interactive. Some techniques produce tools which can automate certain customization operations. Training data, tasks, or document records blend expert-generated, developer-generated, or synthesized items.

Abstract: Recorded sessions of skilled personnel at work are used to train a first machine learning (ML) tool to extract workflow maps. These maps are used to train a second ML tool to emulate at least one workflow. The first ML tool is trained to predict an annotator's output. Either ML tool can be a copilot having a microservice network architecture. Further, complex sets of workflows can be subdivided for efficient support with small ML tools for (1) workflow map extraction and (2) emulation. Based on segment demarcation accompanying a recording, segments are assigned to specialized ML extraction tools for workflow map extraction. Extracted workflow maps are used to train workflow emulator(s). Similarly, specialized ML emulation tools can emulate respective tasks. A distribution microservice can identify a task to be emulated and can invoke the appropriate specialized ML tool. Similar microservice network architectures support both applications.

Abstract: Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. The microservice network architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while significantly reducing hardware requirements and computation time. Disclosed examples incorporate microservices for expansion, retrieval, embedding, and evaluation, in addition to one or more core microservices. Optionally, intermodal I/O, multiple data repositories, competency qualification, or human feedback can be supported. Multiple core microservices can support varying client authorizations or cognitive functions. The disclosed architecture supports any major LLM use case and can be deployed on a single compute node with a single GPU.