Neural Architecture Search (NAS) techniques create complex model architectures by manually searching a smaller portion of the model space. Different NAS algorithms have been proposed and several efficient model architectures have been discovered, including MobileNetV3 and EfficientNet. By reformulating the multi-objective NAS problem within the context of combinatorial optimization, the LayerNAS method significantly reduces the complexity of the problem. This substantially reduces the number of candidate models that need to be searched, the computation required for multi-trial searches, and the identification of best-performing model architectures. Models with top-1 accuracy on ImageNet, up to 4.9% better than existing next-generation alternatives, were discovered using a search space built using backbones derived from MobileNetV2 and MobileNetV3.
LayerNAS relies on search spaces that meet the following two criteria: one of the model options produced by searching the previous layer and using those search options on the current layer can be used to build an ideal model. If the current layer has a FLOP restriction, we can restrict the previous layer by reducing the FLOPs of the current layer. Under these circumstances, it is possible to search linearly from layer 1 to layer n because it is known that changing any previous layer after finding the best match for layer i will not improve model performance.
Candidates can then be grouped based on cost, limiting the number of candidates stored per layer. The most accurate model is only kept when two models have the same FLOPs, as long as doing so does not change the architecture of the lower layers. The layered cost-based approach allows the search space to be significantly reduced while rigorously reasoning about the polynomial complexity of the algorithm. Conversely, to complete the treatment, the search space would increase exponentially with layers because the full range of options is available in each layer. The results of the experimental evaluation show that the best models can be found within these limitations.
LayerNAS reduces NAS to a combinatorial optimization problem by applying a layered cost approach. After training with a specific component Si, the cost and reward can be calculated for each layer i. This implies the following combinatorial problem: How can one choose an option for each layer while staying within a cost budget to achieve the best payoff? There are numerous ways to overcome this problem, but dynamic programming is one of the easiest. The following metrics are evaluated when comparing NAS algorithms: Quality, Stability, and Efficiency. The algorithm is tested against the NATS-Bench benchmark using 100 NAS runs and compared to other NAS algorithms such as random search, regularized evolution, and proximal policy optimization. The differences between these search algorithms are visualized for the metrics described above. The mean precision and precision variance are listed for each comparison (variance is indicated by a shaded rectangle corresponding to the 25% to 75% interquartile range).
To avoid searching for a lot of useless model designs, LayerNAS Performance formulates the problem differently by separating cost and reward. Fewer channels in the earlier layers tend to improve performance in the candidate models. This explains how LayerNAS discovers better models faster than other methods because it doesn’t waste time on models with unfavorable cost distributions. Using combinatorial optimization, which effectively limits the complexity of the search to be polynomial, LayerNAS is proposed as a solution to the multi-target NAS challenge.
The researchers created a new way to find better models for neural networks called LayerNAS. They compared it to other methods and found that it worked better. They also used it to find better models for MobileNetV2 and MobileNetV3.
review the Paper and Reference article. Don’t forget to join our 20k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]
🚀 Check out 100 AI tools at AI Tools Club
Niharika is a technical consulting intern at Marktechpost. She is a third year student, currently pursuing her B.Tech from the Indian Institute of Technology (IIT), Kharagpur. She is a very enthusiastic individual with a strong interest in machine learning, data science, and artificial intelligence and an avid reader of the latest developments in these fields.
