论文标题
无需培训的神经建筑搜索
Neural Architecture Search without Training
论文作者
论文摘要
手工设计深神经网络所涉及的时间和精力是巨大的。这促使了神经体系结构搜索（NAS）技术的开发以自动化此设计。但是，NAS算法往往缓慢而昂贵。他们需要培训大量候选网络以告知搜索过程。如果我们可以从其初始状态中部分预测网络训练有素的准确性，则可以缓解这一点。在这项工作中，我们研究了未经训练网络中数据点之间激活的重叠，并激励它如何提供一种衡量标准，该措施可以用来表明网络训练有素的性能。我们将此度量纳入了一种简单的算法中，该算法使我们可以在单个GPU上几秒钟内搜索强大的网络，而无需进行任何培训，并验证其在NAS-Bench-101，NAS-Bench-2011，NAS-Bench-201，NATS BENCENT和网络设计空间上的有效性。我们的方法很容易与更昂贵的搜索方法相结合；我们检查了正则化进化搜索的简单改编。可以在https://github.com/bayeswatch/nas-without-training上获得复制我们实验的代码。
The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained accuracy from its initial state. In this work, we examine the overlap of activations between datapoints in untrained networks and motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces. Our approach can be readily combined with more expensive search methods; we examine a simple adaptation of regularised evolutionary search. Code for reproducing our experiments is available at https://github.com/BayesWatch/nas-without-training.
