東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Linked to FindBook

Google Book

Amazon

博客來

Posterior Sampling for Efficient Reinforcement Learning.

Record Type:	Electronic resources : Monograph/item
Title/Author:	Posterior Sampling for Efficient Reinforcement Learning./
Author:	Dwaracherla, Vikranth Reddy.
Published:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
Description:	118 p.
Notes:	Source: Dissertations Abstracts International, Volume: 83-09, Section: B.
Contained By:	Dissertations Abstracts International83-09B.
Subject:	Algorithms. -
Online resource:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29003855
ISBN:	9798209788881

Posterior Sampling for Efficient Reinforcement Learning.
Dwaracherla, Vikranth Reddy.

Posterior Sampling for Efficient Reinforcement Learning. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 118 p.

Source: Dissertations Abstracts International, Volume: 83-09, Section: B.

Thesis (Ph.D.)--Stanford University, 2021.

This item must not be sold to any third party vendors.

Reinforcement learning has shown tremendous success over the past few years. Much of this recent success can be attributed to agents learning from an inordinate amount of data in simulated environments. In order to achieve similar success in real environments, it is crucial to address data efficiency. Uncertainty quantification plays a prominent role in designing an intelligent agent which exhibits data efficiency. An agent which has a notion of uncertainty can trade-off between exploration and exploitation and explore in an intelligent manner. Such an agent should not only consider immediate information gain from an action but also its consequences on future learning prospects. An agent which has this capability is said to exhibit deep exploration.Algorithms that tackle deep exploration, so far, have relied on epistemic uncertainty representation through ensembles or other hypermodels, exploration bonuses, or visitation count distributions. An open question is whether deep exploration can be achieved by an incremental reinforcement learning algorithm that tracks a single point estimate, without additional complexity required to account for epistemic uncertainty. We answer this question in the affirmative. In this dissertation, we develop Langevin DQN, a variation of DQN that differs only in perturbing parameter updates with Gaussian noise, and demonstrate through a computational study that Langevin DQN achieves deep exploration. This is the first algorithm that demonstratively achieves deep exploration using a single-point estimate.We also present index sampling, a novel method for efficiently generating approximate samples from a posterior over complex models such as neural networks, induced by a prior distribution over the model family and a set of input-output data pairs. In addition, we develop posterior sampling networks, a new approach to model this distribution over models. We are particularly motivated by the application of our method to tackle reinforcement learning problems, but it could be of independent interest to the Bayesian deep learning community. Our method is especially useful in RL when we use complex exploration schemes, which make use of more than a single sample from the posterior, such as information directed sampling.Finally, we present some preliminary results demonstrating that the Langevin DQN update rule could be used to train posterior sampling networks, as an alternative to index sampling, and further improve data efficiency.

ISBN: 9798209788881Subjects--Topical Terms:

536374
Algorithms.

Posterior Sampling for Efficient Reinforcement Learning.
LDR:03531nmm a2200313 4500 001 2345734
005 20220613063807.5
008 241004s2021 ||||||||||||||||| ||eng d
020 $a 9798209788881
035 $a (MiAaPQ)AAI29003855
035 $a (MiAaPQ)STANFORDjb495qn9584
035 $a AAI29003855
040 $a MiAaPQ $c MiAaPQ
100 1 $a Dwaracherla, Vikranth Reddy. $3 3684728
245 1 0 $a Posterior Sampling for Efficient Reinforcement Learning.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 118 p.
500 $a Source: Dissertations Abstracts International, Volume: 83-09, Section: B.
500 $a Advisor: van Roy, Benjamin;Brunskill, Emma;Pilanci, Mert.
502 $a Thesis (Ph.D.)--Stanford University, 2021.
506 $a This item must not be sold to any third party vendors.
520 $a Reinforcement learning has shown tremendous success over the past few years. Much of this recent success can be attributed to agents learning from an inordinate amount of data in simulated environments. In order to achieve similar success in real environments, it is crucial to address data efficiency. Uncertainty quantification plays a prominent role in designing an intelligent agent which exhibits data efficiency. An agent which has a notion of uncertainty can trade-off between exploration and exploitation and explore in an intelligent manner. Such an agent should not only consider immediate information gain from an action but also its consequences on future learning prospects. An agent which has this capability is said to exhibit deep exploration.Algorithms that tackle deep exploration, so far, have relied on epistemic uncertainty representation through ensembles or other hypermodels, exploration bonuses, or visitation count distributions. An open question is whether deep exploration can be achieved by an incremental reinforcement learning algorithm that tracks a single point estimate, without additional complexity required to account for epistemic uncertainty. We answer this question in the affirmative. In this dissertation, we develop Langevin DQN, a variation of DQN that differs only in perturbing parameter updates with Gaussian noise, and demonstrate through a computational study that Langevin DQN achieves deep exploration. This is the first algorithm that demonstratively achieves deep exploration using a single-point estimate.We also present index sampling, a novel method for efficiently generating approximate samples from a posterior over complex models such as neural networks, induced by a prior distribution over the model family and a set of input-output data pairs. In addition, we develop posterior sampling networks, a new approach to model this distribution over models. We are particularly motivated by the application of our method to tackle reinforcement learning problems, but it could be of independent interest to the Bayesian deep learning community. Our method is especially useful in RL when we use complex exploration schemes, which make use of more than a single sample from the posterior, such as information directed sampling.Finally, we present some preliminary results demonstrating that the Langevin DQN update rule could be used to train posterior sampling networks, as an alternative to index sampling, and further improve data efficiency.
590 $a School code: 0212.
650 4 $a Algorithms. $3 536374
650 4 $a Neural networks. $3 677449
650 4 $a Artificial intelligence. $3 516317
650 4 $a Computer science. $3 523869
690 $a 0800
690 $a 0984
710 2 $a Stanford University. $3 754827
773 0 $t Dissertations Abstracts International $g 83-09B.
790 $a 0212
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29003855