论文标题
监督对比学习作为多目标优化，用于微调大型预训练的语言模型
Supervised Contrastive Learning as Multi-Objective Optimization for Fine-Tuning Large Pre-trained Language Models
论文作者
论文摘要
最近，已证明有监督的对比学习（SCL）可以在大多数分类任务中取得出色的表现。在SCL中，对神经网络进行了训练，可以优化两个目标：在嵌入空间中将锚定和阳性样品一起拉在一起，并将锚点推到负面。但是，这两个不同的目标可能需要冲突，需要在优化期间之间进行权衡。在这项工作中，我们将SCL问题提出为Roberta语言模型的微调阶段的多目标优化问题。使用两种方法来解决优化问题：（i）线性标量（LS）方法，该方法可最大程度地减少持久性损失的加权线性组合； （ii）确切的帕累托最佳（EPO）方法，该方法找到了帕累托前部与给定优先载体的相交。我们在几个粘合基准任务上评估了我们的方法，而无需使用数据增强，内存库或生成对抗性示例。经验结果表明，提出的学习策略大大优于强大的竞争性学习基线
Recently, Supervised Contrastive Learning (SCL) has been shown to achieve excellent performance in most classification tasks. In SCL, a neural network is trained to optimize two objectives: pull an anchor and positive samples together in the embedding space, and push the anchor apart from the negatives. However, these two different objectives may conflict, requiring trade-offs between them during optimization. In this work, we formulate the SCL problem as a Multi-Objective Optimization problem for the fine-tuning phase of RoBERTa language model. Two methods are utilized to solve the optimization problem: (i) the linear scalarization (LS) method, which minimizes a weighted linear combination of pertask losses; and (ii) the Exact Pareto Optimal (EPO) method which finds the intersection of the Pareto front with a given preference vector. We evaluate our approach on several GLUE benchmark tasks, without using data augmentations, memory banks, or generating adversarial examples. The empirical results show that the proposed learning strategy significantly outperforms a strong competitive contrastive learning baseline
