東華大學圖書館 |

語系: 繁體中文

說明(常見問題)

回圖書館首頁

手機版館藏查詢

登入

回首頁

切換: 標籤 | MARC模式 | ISBD

FindBook

Google Book

Amazon

博客來

Vision Based Language to Action Mapping.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Vision Based Language to Action Mapping./
作者:	Shi, Jing.
面頁冊數:	1 online resource (242 pages)
附註:	Source: Dissertations Abstracts International, Volume: 84-11, Section: A.
Contained By:	Dissertations Abstracts International84-11A.
標題:	Computer science. -
電子資源:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29999398click for full text (PQDT)
ISBN:	9798379523558

Vision Based Language to Action Mapping.
Shi, Jing.

Vision Based Language to Action Mapping. - 1 online resource (242 pages)

Source: Dissertations Abstracts International, Volume: 84-11, Section: A.

Thesis (Ph.D.)--University of Rochester, 2022.

Includes bibliographical references

Vision and language are two major modalities in the world we perceive. In academia, they have spawned a large body of research problems, such as cross-modal matching (image text retrieval, visual grounding), cross-modal reasoning (visual question answering, visual common sense reasoning), vision and text generation (text-to-image generation and manipulation), etc. Prevalent research addresses these problems via holistic modeling, which only fuses the extracted visual and textual embedding and yields the output through an output layer. Although holistic modeling such as VisualBert and CLIP has achieved promising performance, the compositional reasoning ability is still limited due to the vast learning space, and the inference process is human-uninterpretable ascribed to the black-box modeling. Therefore, we propose to model the vision and language problems by decomposing the reasoning process into step-by-step interpretable actions, where learning and planning are also involved. Such neural-symbolic style modeling allows interpretable reasoning, injection of prior knowledge, reduction of model parameter space, alleviation of learning difficulty, and regularization of the inference process.The major challenges of introducing the action into more detailed vision-language modeling are (i) limited visual perception ability on the dataset without object location annotation and (ii) difficulty generating actions without intermediate action supervision. Therefore, this thesis proposes to address these two challenges in two parts. The first part presents our solutions to align the visual instance to the textual concept via vision and language matching, allowing the learning of grounding, detection, and relation, given only the week supervision of image-sentence alignment. The second part introduces language as action to generate action without intermediate action labels on the language-guided image editing task, pioneering a new perspective on the cross-modal visual generation/manipulation problem. We prove that our solution can generalize to unconstrained images and open-vocabulary language.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2023

Mode of access: World Wide Web

ISBN: 9798379523558Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Scene graph generationIndex Terms--Genre/Form:

542853
Electronic books.

Vision Based Language to Action Mapping.
LDR:03531nmm a2200409K 4500 001 2358366
005 20230731112630.5
006 m o d
007 cr mn ---uuuuu
008 241011s2022 xx obm 000 0 eng d
020 $a 9798379523558
035 $a (MiAaPQ)AAI29999398
035 $a AAI29999398
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Shi, Jing. $3 3184666
245 1 0 $a Vision Based Language to Action Mapping.
264 0 $c 2022
300 $a 1 online resource (242 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 84-11, Section: A.
500 $a Advisor: Xu, Chenliang; Li, Dongmei.
502 $a Thesis (Ph.D.)--University of Rochester, 2022.
504 $a Includes bibliographical references
520 $a Vision and language are two major modalities in the world we perceive. In academia, they have spawned a large body of research problems, such as cross-modal matching (image text retrieval, visual grounding), cross-modal reasoning (visual question answering, visual common sense reasoning), vision and text generation (text-to-image generation and manipulation), etc. Prevalent research addresses these problems via holistic modeling, which only fuses the extracted visual and textual embedding and yields the output through an output layer. Although holistic modeling such as VisualBert and CLIP has achieved promising performance, the compositional reasoning ability is still limited due to the vast learning space, and the inference process is human-uninterpretable ascribed to the black-box modeling. Therefore, we propose to model the vision and language problems by decomposing the reasoning process into step-by-step interpretable actions, where learning and planning are also involved. Such neural-symbolic style modeling allows interpretable reasoning, injection of prior knowledge, reduction of model parameter space, alleviation of learning difficulty, and regularization of the inference process.The major challenges of introducing the action into more detailed vision-language modeling are (i) limited visual perception ability on the dataset without object location annotation and (ii) difficulty generating actions without intermediate action supervision. Therefore, this thesis proposes to address these two challenges in two parts. The first part presents our solutions to align the visual instance to the textual concept via vision and language matching, allowing the learning of grounding, detection, and relation, given only the week supervision of image-sentence alignment. The second part introduces language as action to generate action without intermediate action labels on the language-guided image editing task, pioneering a new perspective on the cross-modal visual generation/manipulation problem. We prove that our solution can generalize to unconstrained images and open-vocabulary language.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2023
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 523869
650 4 $a Language. $3 643551
650 4 $a Computer engineering. $3 621879
653 $a Scene graph generation
653 $a Scene understanding
653 $a Visual Synthesis/Manipulation
653 $a Holistic modeling
653 $a Text generation
653 $a Neural-symbolic style
655 7 $a Electronic books. $2 lcsh $3 542853
690 $a 0984
690 $a 0464
690 $a 0679
710 2 $a ProQuest Information and Learning Co. $3 783688
710 2 $a University of Rochester. $b Hajim School of Engineering and Applied Sciences. $3 2099687
773 0 $t Dissertations Abstracts International $g 84-11A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29999398 $z click for full text (PQDT)