Abstract: Systems and methods for generating a small language model are provided. In particular, a computing device may obtain a general dataset including a plurality of general data, annotate a subset of the general dataset based on one or more classifier metrics indicative of a quality of the general dataset, train a classifier based on the annotated subset of the general dataset and the one or more classifier metrics, analyze each general data of the general dataset to determine a score for each of the one or more classifier metrics associated with the respective general data using the trained classifier, generate a filtered general dataset by filtering the general dataset based on one or more filters, train the small language model with the filtered general dataset, generate a synthetic dataset for refining the small language model, and train the small language model with the synthetic dataset.

Abstract: In some embodiments, there is provided a system, which comprises a processor, and at least one non-transitory computer readable media storing instructions.

Abstract: A computing system determines a median of a first number of mean values received from a first number of clusters, where each cluster of the first number of clients includes a first plurality of clients. Also, a threshold is determined based on the median, where the threshold applies to model updates. The median and the threshold are broadcast to all clients. Next, one or more clients that fail to provide a proof attesting that their model update is within the threshold of the median are dropped. Then, a second plurality of clients, not including the one or more dropped clients, participate in a final round of secure aggregation. Next, a final aggregate result is obtained, where the final aggregate result is based on the final round of secure aggregation. Then, one or more actions are performed based on the final aggregate result.

Abstract: Generally discussed herein are devices, systems, and methods for improving architecture search and identification with constraints. A method can include receiving, at a compute device, a request for a transformer-based autoregressive language model (TBALM), the request specifying a maximum latency, identifying TBALM architectures that satisfies the maximum latency, identifying a TBALM architecture of the identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, and providing the identified TBALM architecture.

Abstract: A method, a system, and a computer program product for fast training and/or execution of neural networks. A description of a neural network architecture is received. Based on the received description, a graph representation of the neural network architecture is generated. The graph representation includes one or more nodes connected by one or more connections. At least one connection is modified. Based on the generated graph representation, a new graph representation is generated using the modified at least one connection. The new graph representation has a small-world property. The new graph representation is transformed into a new neural network architecture.

Abstract: A method, a system, and a computer program product for fast training and/or execution of neural networks. A description of a neural network architecture is received. Based on the received description, a graph representation of the neural network architecture is generated. The graph representation includes one or more nodes connected by one or more connections. At least one connection is modified. Based on the generated graph representation, a new graph representation is generated using the modified at least one connection. The new graph representation has a small-world property. The new graph representation is transformed into a new neural network architecture.