東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Linked to FindBook

Google Book

Amazon

博客來

Understanding Deep Learning via Analyzing Dynamics of Gradient Descent.

Record Type:	Electronic resources : Monograph/item
Title/Author:	Understanding Deep Learning via Analyzing Dynamics of Gradient Descent./
Author:	Hu, Wei.
Published:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
Description:	340 p.
Notes:	Source: Dissertations Abstracts International, Volume: 83-04, Section: B.
Contained By:	Dissertations Abstracts International83-04B.
Subject:	Computer science. -
Online resource:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28714897
ISBN:	9798471108073

Understanding Deep Learning via Analyzing Dynamics of Gradient Descent.
Hu, Wei.

Understanding Deep Learning via Analyzing Dynamics of Gradient Descent. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 340 p.

Source: Dissertations Abstracts International, Volume: 83-04, Section: B.

Thesis (Ph.D.)--Princeton University, 2021.

This item must not be sold to any third party vendors.

The phenomenal successes of deep learning build upon the mysterious abilities of gradient-based optimization algorithms. Not only can these algorithms often successfully optimize complicated non-convex training objectives, but the solutions found can also generalize remarkably well to unseen test data despite significant over-parameterization of the models. Classical approaches in optimization and generalization theories that treat empirical risk minimization as a black box are insufficient to explain these mysteries in modern deep learning. This dissertation illustrates how we can make progress toward understanding optimization and generalization in deep learning by a more refined approach that opens the black box and analyzes the dynamics taken by the optimizer. In particular, we present several theoretical results that take into account the learning dynamics of the gradient descent algorithm.In the first part, we provide global convergence guarantees of gradient descent for training deep linear networks under various initialization schemes. Our results characterize the effect of width, depth and initialization on the speed of optimization. In addition, we identify an auto-balancing effect of gradient flow, which we prove to hold generally in homogeneous neural networks (including those with ReLU activation).In the second part, we study the implicit regularization induced by gradient descent, which is believed to be the key to mathematically understanding generalization in deep learning. We present results in both linear and non-linear neural networks, which characterize how gradient descent implicitly favors simple solutions.In the third part, we focus on the setting where neural networks are over-parameterized to have sufficiently large width. Through the connection to neural tangent kernels, we perform a fine-grained analysis of optimization and generalization, which explains several empirically observed phenomena. Built on these theoretical principles, we further design a new simple and effective method for training neural networks on noisily labeled data.

ISBN: 9798471108073Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Generalization

Understanding Deep Learning via Analyzing Dynamics of Gradient Descent.
LDR:03213nmm a2200361 4500 001 2348076
005 20220906075159.5
008 241004s2021 ||||||||||||||||| ||eng d
020 $a 9798471108073
035 $a (MiAaPQ)AAI28714897
035 $a AAI28714897
040 $a MiAaPQ $c MiAaPQ
100 1 $a Hu, Wei. $3 1043762
245 1 0 $a Understanding Deep Learning via Analyzing Dynamics of Gradient Descent.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 340 p.
500 $a Source: Dissertations Abstracts International, Volume: 83-04, Section: B.
500 $a Advisor: Arora, Sanjeev.
502 $a Thesis (Ph.D.)--Princeton University, 2021.
506 $a This item must not be sold to any third party vendors.
520 $a The phenomenal successes of deep learning build upon the mysterious abilities of gradient-based optimization algorithms. Not only can these algorithms often successfully optimize complicated non-convex training objectives, but the solutions found can also generalize remarkably well to unseen test data despite significant over-parameterization of the models. Classical approaches in optimization and generalization theories that treat empirical risk minimization as a black box are insufficient to explain these mysteries in modern deep learning. This dissertation illustrates how we can make progress toward understanding optimization and generalization in deep learning by a more refined approach that opens the black box and analyzes the dynamics taken by the optimizer. In particular, we present several theoretical results that take into account the learning dynamics of the gradient descent algorithm.In the first part, we provide global convergence guarantees of gradient descent for training deep linear networks under various initialization schemes. Our results characterize the effect of width, depth and initialization on the speed of optimization. In addition, we identify an auto-balancing effect of gradient flow, which we prove to hold generally in homogeneous neural networks (including those with ReLU activation).In the second part, we study the implicit regularization induced by gradient descent, which is believed to be the key to mathematically understanding generalization in deep learning. We present results in both linear and non-linear neural networks, which characterize how gradient descent implicitly favors simple solutions.In the third part, we focus on the setting where neural networks are over-parameterized to have sufficiently large width. Through the connection to neural tangent kernels, we perform a fine-grained analysis of optimization and generalization, which explains several empirically observed phenomena. Built on these theoretical principles, we further design a new simple and effective method for training neural networks on noisily labeled data.
590 $a School code: 0181.
650 4 $a Computer science. $3 523869
650 4 $a Artificial intelligence. $3 516317
650 4 $a Applied mathematics. $3 2122814
653 $a Generalization
653 $a Learning theory
653 $a Machine learning
653 $a Optimization
690 $a 0984
690 $a 0800
690 $a 0364
710 2 $a Princeton University. $b Computer Science. $3 2099280
773 0 $t Dissertations Abstracts International $g 83-04B.
790 $a 0181
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28714897