论文标题
学生需要更多的关注：针对小数据的基于BERT的注意力模型，并应用于自动化消息Triage
Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage
论文作者
论文摘要
在基于深度学习模型培训分类器时，医疗保健中常见的小而失衡的数据集代表了一个挑战。我们如此积极，我们提出了一个基于生物Biobert的新型框架（来自变形金刚的双向编码器表示，而传统的文本编码）。具体而言，（i）我们将标签嵌入在Bert的每一层中进行自我注意，我们称为Lesa-Bert，（ii）（ii）通过将Lesa-Bert提炼为较小的变体，我们的目标是在小型数据集上工作时降低过度拟合和模型大小。作为一个应用程序，我们的框架被用来为患者门户网站消息分类建立模型，将消息的紧迫性分为三类：非紧急，中等和紧急。实验表明，就宏F1分数而言，我们的方法可以比几个强大的基线分类器的表现高4.3％。该项目的代码可在\ url {https://github.com/shijing001/text_classifiers}上公开获得。
Small and imbalanced datasets commonly seen in healthcare represent a challenge when training classifiers based on deep learning models. So motivated, we propose a novel framework based on BioBERT (Bidirectional Encoder Representations from Transformers forBiomedical TextMining). Specifically, (i) we introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent. Experiments demonstrate that our approach can outperform several strong baseline classifiers by a significant margin of 4.3% in terms of macro F1 score. The code for this project is publicly available at \url{https://github.com/shijing001/text_classifiers}.
