论文标题
复发性神经切线内核
The Recurrent Neural Tangent Kernel
论文作者
论文摘要
通过所谓的神经切线核（NTK）方法，深度神经网络（DNNS）的研究为学习，概括和初始化的影响提供了新的见解。一个关键的DNN体系结构仍有待核，即经常性神经网络（RNN）。在本文中，我们介绍并研究了复发性神经切线核（RNTK），该神经切线核（RNTK）提供了对过多散热性RNN的行为的新见解。 RNTK的关键属性应极大地使从业人员受益于其比较不同长度的输入的能力。为此，我们表征了RNTK如何在不同的初始化参数和非线性选择下加权不同的时间步骤以形成其输出。合成和56个现实世界数据实验表明，RNTK在广泛的数据集上比其他内核（包括标准NTK）提供了显着的性能增长。
The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.
