Abstract: Methods and systems are described herein for evaluating SFT processes for large language models (LLMs). For example, a request to evaluate a performance of a trained large language model to execute a computing task may be received. The request may indicate an evaluation data set including a plurality of evaluation samples. A prompt template may be retrieved based on the request. For each evaluation sample of the plurality of evaluation samples, the system may generate a sample prompt by incorporating a query and context from an evaluation sample into the prompt template. The sample prompt may be input into the large language model to obtain a sample evaluation result. An evaluation metric score may be determined based on the sample evaluation result and a reference evaluation result associated with the evaluation sample. A model performance score may be generated based on the evaluation score for each evaluation sample.

Abstract: A method and related system may reduce hallucinatory text generation by using shuffled training data.

Abstract: Methods and systems are described herein for managing prompts used for executing and evaluating supervised fine tuning (SFT) processes. For example, a model identifier and a first description of a first computing task may be extracted from a first request to execute an SFT process to perform the first computing task. A candidate prompt template and associated metadata including a candidate prompt name may be generated, and the candidate prompt template may be uploaded to a prompt database with the associated metadata. A second request to execute an SFT process indicating the candidate prompt name may be received. The candidate prompt template may be used to execute the SFT process on a large language model associated with the second request, resulting in a trained first large language model. The trained first large language model may be evaluated using a test data set and the candidate prompt template.

Abstract: Methods and systems are described herein for optimizing the fine-tuning and evaluation thereof for large language models (LLMs). For example, a request to (i) execute a supervised fine tuning (SFT) process to train a large language model and (ii) evaluate a performance of the large language model after execution can be received. The request can indicate (a) a training data set to be used for the SFT process, (b) a test data set to be used to evaluate the performance of the large language model, and (c) a set of evaluation metrics to evaluate the performance of the large language model after the SFT process has been executed. A trained model can be obtained based on executing the SFT process using the training data set and a prompt template. A set of evaluation metric scores can be obtained based on evaluating the trained model using the test data set.

Abstract: Computer-readable media, methods, and systems are disclosed for optimizing processing across a plurality of processing resources using one or more leader threads to assign processing tasks to available processing threads across a plurality of application instances. The one or more leader threads monitor the status and availability of the processing threads and a processing load across the plurality of application instances to efficiently assign processing tasks and distribute the processing load across the processing threads of the application instances.

Abstract: A device (102) for lighting automation includes at least one communication interface (122, 124) corresponding to a plurality of lighting fixtures (130, 132), wherein each of the lighting fixture is associated with a corresponding physical interface (122, 124), the device is configured to detect a gesture (106) by a user (108), via a mobile device (104), to establish an association with a physical interface. The device (102) is further configured to detect a lighting fixture (130, 132) controllable by the physical interface (122, 124) and generate control signals (120) for the detected lighting fixture, via an orientation sensor, based on movements of the mobile device. The device (102) is also configured to operate the lighting fixture corresponding to the physical interface based on the generated control signals. The movements of the mobile device (104) are similar to the movements involved in the operation of the physical interface (122, 124).

Abstract: Systems and methods for improved data processing of communications across computer networks using trifurcated prompts during communication exchanges are described. For example, the system may receive a first inbound communication, wherein the first inbound communication system may determine a first context for the first inbound communication based on the first text string. The system may process the first context in a perturbation model to determining a first perturbed context, wherein the perturbation model determines the first perturbed context by determining a first alternative token for a first token in the first context. The system may determine a first prompt for a first large language model based on the first perturbed context.

Abstract: This invention relates to overbased calcium sulfonate complex grease compositions prepared without boric acid and alcohol promotors containing anti-wear, anti-friction, thermal & oxidation stability additives.

Abstract: Techniques for generating a natural language prompt to further a goal of a dialog, are described. During a dialog, the system receives one or more user inputs including a user question, a user response to the question, and a request to generate a further question following the response. The system determines ASR output data corresponding to the user inputs, and determines dialog history data of the dialog. Using the ASR output data and the dialog history data, the system determines a category and an explanation of relevance corresponding to the category. Using the ASR output data, the dialog history, the category, and the explanation, the system determines the further question to be output to the user.

Abstract: Techniques for updating a large language model (LLM) to correct generation of undesired responses, such as incorrect outputs, toxic outputs, etc. are described. Typical methods of retraining and fine-tuning are inefficient and computationally expensive for LLMs. Some embodiments of the present disclosure involve identifying a salient layer of the LLM that is responsible for the undesired response and editing only the salient layer. This layer is identified by computing a saliency value for the layer using a mean of gradient values for the layer, and the layer with the greatest saliency value is selected for editing. For editing, a small network is used to update the weights of the selected layer. The LLM is updated to include the edited layer, and the updated LLM is used for future processing.

Abstract: The present disclosure relates to systems and methods for a system lab and SOP framework to support testing based on customer environments. Longevity testing must be run on every build with different test cases; thus, the system lab framework allows this testing to be performed on multiple machines in a single run, also allowing test results to be stored. The system lab framework is adapted to recreate customer issues, allowing different environments typically not accessible to be replicated and tested. The system lab framework includes central code to trigger cases on different machines. A central data collection system is able to collect all of the results from the cases and store the respective data. This system lab framework is adapted to support multiple Operating Systems (OS) including Windows, Mac, Linux, and the like.

Abstract: Techniques for evaluating a natural language understanding (NLU) component and determining an action to resolve an issue processing a user input are described. The system determines which component is invoked by a baseline NLU component is processing the user input, and which component is invoked by an updated NLU component. Based on that information, the system selects the action to resolve the updated NLU component generating an undesired response to the user input.

Abstract: Systems and methods are disclosed to address inter-cell interference in a heterogeneous network. In one embodiment, a system is disclosed, comprising: a coordinating node situated between a radio access network and a core network; and a first base station in the radio access network in communication with the coordinating node, wherein: the coordinating node has a coordinating scheduler with a first scheduling period; the first base station has a first base station scheduler with a second scheduling period shorter than the first scheduling period; the coordinating scheduler is configured to send a resource reservation list and a resource restriction list to the first base station scheduler once during each first scheduling period; and the first base station is configured to receive the resource reservation list and the resource restriction list and to use the resource reservation list and the resource restriction list when performing mobile device resource scheduling.

Abstract: The present disclosure relates to systems and methods for a system lab and SOP framework to support testing based on customer environments. Longevity testing must be run on every build with different test cases; thus, the system lab framework allows this testing to be performed on multiple machines in a single run, also allowing test results to be stored. The system lab framework is adapted to recreate customer issues, allowing different environments typically not accessible to be replicated and tested. The system lab framework includes central code to trigger cases on different machines. A central data collection system is able to collect all of the results from the cases and store the respective data. This system lab framework is adapted to support multiple Operating Systems (OS) including Windows, Mac, Linux, and the like.

Abstract: A method of providing user level mobility load balancing is provided, the method comprising: classifying a group of User Equipments (UEs) in a cell into multidimensional planes based on metrics associated with the UEs; defining thresholds for each dimension; using the defined thresholds determining to discard or select certain planes of a dimension and the UEs contained in the planes; and identifying a best UE of the UEs contained in selected planes for offload.

Abstract: This invention relates to overbased calcium sulfonate complex grease compositions prepared without boric acid and alcohol promotors containing anti-wear, anti-friction, thermal & oxidation stability additives.

Abstract: Techniques for evaluating a natural language understanding (NLU) component and determining an action to resolve an issue processing a user input are described. The system determines which component is invoked by a baseline NLU component is processing the user input, and which component is invoked by an updated NLU component. Based on that information, the system selects the action to resolve the updated NLU component generating an undesired response to the user input.

Abstract: Techniques for using a federated learning framework to update machine learning models for spoken language understanding (SLU) system are described. The system determines which labeled data is needed to update the models based on the models generating an undesired response to an input. The system identifies users to solicit labeled data from, and sends a request to a user device to speak an input. The device generates labeled data using the spoken input, and updates the on-device models using the spoken input and the labeled data. The updated model data is provided to the system to enable the system to update the system-level (global) models.

Abstract: Techniques for evaluating a natural language understanding (NLU) component and determining an action to resolve an issue processing a user input are described. The system determines which component is invoked by a baseline NLU component is processing the user input, and which component is invoked by an updated NLU component. Based on that information, the system selects the action to resolve the updated NLU component generating an undesired response to the user input.

Abstract: Systems and methods are disclosed to address inter-cell interference in a heterogeneous network. In one embodiment, a system is disclosed, comprising: a coordinating node situated between a radio access network and a core network; and a first base station in the radio access network in communication with the coordinating node, wherein: the coordinating node has a coordinating scheduler with a first scheduling period; the first base station has a first base station scheduler with a second scheduling period shorter than the first scheduling period; the coordinating scheduler is configured to send a resource reservation list and a resource restriction list to the first base station scheduler once during each first scheduling period; and the first base station is configured to receive the resource reservation list and the resource restriction list and to use the resource reservation list and the resource restriction list when performing mobile device resource scheduling.