论文标题
更大的章鱼是否仍然会扩大报告偏见？典型颜色判断的证据
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour
论文作者
论文摘要
在原始文本上培训的语言模型（LMS）无法直接访问物理世界。 Gordon和Van Durme（2013）指出，LMS因此可能会遭受报告偏见的困扰：文本很少报告常见事实，而是关注情况的异常方面。如果LMS仅接受文本语料库的培训并天真地记住当地的同时统计数据，那么他们自然会学会对物理世界的有偏见。虽然先前的研究反复验证了较小规模的LM（例如Roberta，GPT-2）放大了报告偏差，但在模型扩展时，此类趋势是否继续。我们从较大语言模型（LLM）（例如Palm和GPT-3）中从颜色的角度研究报告偏见。具体而言，我们查询llms的典型物体颜色，这是一种简单的感知扎根的物理常识。令人惊讶的是，我们发现LLM在确定对象的典型颜色和更紧密地跟踪人类判断方面的表现明显优于较小的LMS，而不是过度适合存储在文本中的表面图案。这表明，仅凭语言的大型语言模型就能克服以局部共发生为特征的某些类型的报告偏差。
Language models (LMs) trained on raw texts have no direct access to the physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer from reporting bias: texts rarely report on common facts, instead focusing on the unusual aspects of a situation. If LMs are only trained on text corpora and naively memorise local co-occurrence statistics, they thus naturally would learn a biased view of the physical world. While prior studies have repeatedly verified that LMs of smaller scales (e.g., RoBERTa, GPT-2) amplify reporting bias, it remains unknown whether such trends continue when models are scaled up. We investigate reporting bias from the perspective of colour in larger language models (LLMs) such as PaLM and GPT-3. Specifically, we query LLMs for the typical colour of objects, which is one simple type of perceptually grounded physical common sense. Surprisingly, we find that LLMs significantly outperform smaller LMs in determining an object's typical colour and more closely track human judgments, instead of overfitting to surface patterns stored in texts. This suggests that very large models of language alone are able to overcome certain types of reporting bias that are characterized by local co-occurrences.
