Abstract: Generally discussed herein are devices, systems, and methods for improving architecture search and identification with constraints. A method can include receiving, at a compute device, a request for a transformer-based autoregressive language model (TBALM), the request specifying a maximum latency, identifying TBALM architectures that satisfies the maximum latency, identifying a TBALM architecture of the identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, and providing the identified TBALM architecture.

Abstract: A method, a system, and a computer program product for fast training and/or execution of neural networks. A description of a neural network architecture is received. Based on the received description, a graph representation of the neural network architecture is generated. The graph representation includes one or more nodes connected by one or more connections. At least one connection is modified. Based on the generated graph representation, a new graph representation is generated using the modified at least one connection. The new graph representation has a small-world property. The new graph representation is transformed into a new neural network architecture.

Abstract: A method, a system, and a computer program product for fast training and/or execution of neural networks. A description of a neural network architecture is received. Based on the received description, a graph representation of the neural network architecture is generated. The graph representation includes one or more nodes connected by one or more connections. At least one connection is modified. Based on the generated graph representation, a new graph representation is generated using the modified at least one connection. The new graph representation has a small-world property. The new graph representation is transformed into a new neural network architecture.