東華大學圖書館 |

語系: 繁體中文

說明(常見問題)

回圖書館首頁

手機版館藏查詢

登入

回首頁

切換: 標籤 | MARC模式 | ISBD

Analyzing and Improving Compositiona...

Yu, Lang.

FindBook

Google Book

Amazon

博客來

Analyzing and Improving Compositionality in Neural Language Models = = 分析和改善神经语言模型的组成性.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Analyzing and Improving Compositionality in Neural Language Models =/
其他題名:	分析和改善神经语言模型的组成性.
作者:	Yu, Lang.
出版者:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
面頁冊數:	109 p.
附註:	Source: Dissertations Abstracts International, Volume: 83-01, Section: B.
Contained By:	Dissertations Abstracts International83-01B.
標題:	Computer science. -
電子資源:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28419838
ISBN:	9798516957031

Analyzing and Improving Compositionality in Neural Language Models = = 分析和改善神经语言模型的组成性.
Yu, Lang.

Analyzing and Improving Compositionality in Neural Language Models =分析和改善神经语言模型的组成性. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 109 p.

Source: Dissertations Abstracts International, Volume: 83-01, Section: B.

Thesis (Ph.D.)--The University of Chicago, 2021.

This item must not be sold to any third party vendors.

Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs. However, we have limited understanding of how these models handle representation of input sequences, and whether this reflects sophisticated composition of meaning like that done by humans. In this dissertation, we take steps to analyze and improve compositionality in natural language models.We present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.Motivated by the observations of pre-trained transformers, we explore directions of improving compositionality in neural language models. We first investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information beyond lexical content. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. After fine-tuning, we assess phrasal representations in controlled settings following prior work. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models. In follow-up analyses, we identify confounding cues in the paraphrase dataset that may explain the lack of composition benefits from that task, and we discuss factors underlying the localized benefits from sentiment training. We then inspect a model with compositional architecture and show that the model shows weak compositionality despite incorporating explicit composition structure.

ISBN: 9798516957031Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Artificial intelligence

Analyzing and Improving Compositionality in Neural Language Models = = 分析和改善神经语言模型的组成性.
LDR:05830nmm a2200397 4500 001 2283136
005 20211022115652.5
008 220723s2021 ||||||||||||||||| ||eng d
020 $a 9798516957031
035 $a (MiAaPQ)AAI28419838
035 $a AAI28419838
040 $a MiAaPQ $c MiAaPQ
100 1 $a Yu, Lang. $3 3276967
245 1 0 $a Analyzing and Improving Compositionality in Neural Language Models = $b 分析和改善神经语言模型的组成性.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 109 p.
500 $a Source: Dissertations Abstracts International, Volume: 83-01, Section: B.
500 $a Advisor: Ettinger, Allyson.
502 $a Thesis (Ph.D.)--The University of Chicago, 2021.
506 $a This item must not be sold to any third party vendors.
520 $a Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs. However, we have limited understanding of how these models handle representation of input sequences, and whether this reflects sophisticated composition of meaning like that done by humans. In this dissertation, we take steps to analyze and improve compositionality in natural language models.We present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.Motivated by the observations of pre-trained transformers, we explore directions of improving compositionality in neural language models. We first investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information beyond lexical content. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. After fine-tuning, we assess phrasal representations in controlled settings following prior work. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models. In follow-up analyses, we identify confounding cues in the paraphrase dataset that may explain the lack of composition benefits from that task, and we discuss factors underlying the localized benefits from sentiment training. We then inspect a model with compositional architecture and show that the model shows weak compositionality despite incorporating explicit composition structure.
520 $a 深层transformer模型近年来已经不断将自然语言处理（NLP）应用的性能推向新高，这表明这些模型能够成熟而精细地处理复杂的语言输入。然而，对于这些模型是如何生成针对输入文本的向量的过程，现有的研究对此理解十分有限。并且，对于这种精细的语义生成过程（composition of meaning），我们尚未确定是否类似于人类对语义的处理。在这篇论文中，我们将分步研究如何分析和改善神经语言模型的组成性（compositionality）。我们首先讨论了针对预训练（pre-trained）transformer模型的短语向量（phrasal representations）的系统分析。我们使用人类对于短语相似性的判断和语义变化来进行测试，并进一步比较在控制和不控制短语中词重叠量时模型的性能，以此将词汇记忆和词义理解在向量生成过程中的影响剥离。通过系统分析，我们发现在这些模型中，短语向量的生成极大程度上依赖词内容，而非语义的组成。我们同时发现短语向量的质量在不同模型，不同网络层和不同向量类型中有变化。对此，我们根据这些模型中不同向量的用途做出有针对性的建议。基于我们对预训练transformer模型的分析，我们进一步探索了改善神经语言模型组成性的方法。我们首先研究了fine-tuning对contextualized embeddings在包含词汇内容以外，表征短语语义信息的影响。具体地，我们使用含有高词重叠量的对抗复述分类（adversarial paraphrase classification）数据集和情感分类数据集（sentiment classification）来分别fine-tune这些模型。在进行fine-tuning之后，我们使用前文提出的评价任务集来对短语向量进行评估。我们发现，fine-tuning在大多数情况下并不能改善这些向量的组成性，但是在情感数据集上训练的特定模型获得了略微的改善。对此，我们进行了进一步分析，我们发现存在于对抗数据集中的一些线索可能是导致训练后模型缺乏组成性的原因。同时，我们讨论了情感数据集带来性能收益的底层因素。我们接下来分析了一个包含显式组成结构的模型，并发现尽管此模型包含了组成结构，但仍然只表现出较弱的组成性。.
590 $a School code: 0330.
650 4 $a Computer science. $3 523869
650 4 $a Linguistics. $3 524476
650 4 $a Language. $3 643551
650 4 $a Internships. $3 3560137
650 4 $a Accuracy. $3 3559958
650 4 $a Datasets. $3 3541416
650 4 $a Sentiment analysis. $2 lcstt $3 3266790
650 4 $a Human performance. $3 3562051
650 4 $a Experiments. $3 525909
650 4 $a Classification. $3 595585
650 4 $a Dissertations & theses. $3 3560115
650 4 $a Advisors. $3 3560734
650 4 $a Natural language. $3 3562052
653 $a Artificial intelligence
653 $a Computational linguistics
653 $a Interpretability and analysis
653 $a Language modeling
653 $a Machine learning
653 $a Natural language processing
690 $a 0984
690 $a 0290
690 $a 0679
710 2 $a The University of Chicago. $b Computer Science. $3 1674744
773 0 $t Dissertations Abstracts International $g 83-01B.
790 $a 0330
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28419838