Abstract: Systems and methods for performing neural architecture search are provided. In one aspect, the system includes a processor configured to select a plurality of candidate neural networks within a search space, evaluate a performance of each of the plurality of candidate neural networks by: training each candidate neural network on a training dataset to perform the predetermined task and determining a ranking metric for each candidate neural network based on an objective function. The ranking metric includes a weight-related metric that is determined based on weights of a prediction layer of each respective candidate neural network before and after the respective candidate neural network is trained. The processor is configured to rank the plurality of candidate neural networks based on the determined ranking metrics.