Abstract: The subject technology receives code corresponding to a neural network (NN) model and a set of weights for the NN model. The subject technology determines a set of layers that are mutable in the NN model. The subject technology determines information for mapping a second set of weights to the set of weights for the NN model. The subject technology generates metadata corresponding to the set of layers that are mutable, and the information for mapping the second set of weights to the set of weights for the NN model, wherein the generated metadata enables updating the set of layers that are mutable during execution of the NN model.

Abstract: An electronic device may receive, at a first system routine from a client application, a provisioning request indicating that the application includes code for evaluating a machine learning model, wherein the first system routine executes in user space of memory on the device. The device may provision the code for execution on one or more of the circuit engines. The device may receive, at a second system routine from the application, an inference request for evaluating the machine learning model containing input data, wherein the second system routine executes in kernel space of memory on the device. The device may receive, at the second system routine, information about the circuit engines. The device may assign the inference request to one or more of the circuit engines where the request is evaluated. The device may provide a result of the inference request to the application.

Abstract: The subject technology receives code corresponding to a neural network (NN) model, the code including particular operations that are performed by the NN model. The subject technology determines, among the particular operations, a set of operations that are to be allocated to a cache of the electronic device that is to execute the NN model. The subject technology generates a set of cache indicators corresponding to the determined set of operations. The subject technology compiles the code and the generated set of cache indicators to provide a compiled binary for the NN model to execute on a target device.

Abstract: The subject technology receives code corresponding to a neural network (NN) model, the code including particular operations that are performed by the NN model. The subject technology determines, among the particular operations, a set of operations that are to be allocated to a cache of the electronic device that is to execute the NN model. The subject technology generates a set of cache indicators corresponding to the determined set of operations. The subject technology compiles the code and the generated set of cache indicators to provide a compiled binary for the NN model to execute on a target device.

Abstract: The subject technology receives code corresponding to a neural network (NN) model and a set of weights for the NN model. The subject technology determines a set of layers that are mutable in the NN model. The subject technology determines information for mapping a second set of weights to the set of weights for the NN model. The subject technology generates metadata corresponding to the set of layers that are mutable, and the information for mapping the second set of weights to the set of weights for the NN model, wherein the generated metadata enables updating the set of layers that are mutable during execution of the NN model.