Abstract: Provided are systems and methods that improve the computational efficiency of Transformers or other attention-based neural networks or machine learning models by re-using a number of attention scores between layers and/or heads of the model. To reduce the computational cost of self-attention-based models while achieving comparable or even superior results, example aspects of the present disclosure propose a novel architecture that reuses attention scores computed in one layer in one or multiple subsequent layers.

Abstract: A neural network based learning system for designing a circuit, the design system including at least one memory, at least one processor in communication with said at least one memory, said at least one processor configured to generate a mathematical model of the circuit, determine a structural definition of the circuit from the mathematical model, define a mapping of a plurality of components of the circuit to a plurality of neurons representing the plurality of components of the circuit using at least the structural definition, synthesize, on a hardware substrate, the plurality of neurons, and execute, using the synthesized plurality of neurons on the hardware substrate, at least one test using at least one optimization constraint to determine an optimal arrangement of the plurality of components.