東華大學圖書館 |

語系: 繁體中文

說明(常見問題)

回圖書館首頁

手機版館藏查詢

登入

回首頁

切換: 標籤 | MARC模式 | ISBD

FindBook

Google Book

Amazon

博客來

Analysis of Quantization and Normalization Effects in Deep Neural Networks.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Analysis of Quantization and Normalization Effects in Deep Neural Networks./
作者:	Chai, Elaina Teresa.
出版者:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
面頁冊數:	153 p.
附註:	Source: Dissertations Abstracts International, Volume: 83-05, Section: B.
Contained By:	Dissertations Abstracts International83-05B.
標題:	Neurons. -
電子資源:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28812913
ISBN:	9798494462435

Analysis of Quantization and Normalization Effects in Deep Neural Networks.
Chai, Elaina Teresa.

Analysis of Quantization and Normalization Effects in Deep Neural Networks. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 153 p.

Source: Dissertations Abstracts International, Volume: 83-05, Section: B.

Thesis (Ph.D.)--Stanford University, 2021.

This item must not be sold to any third party vendors.

There is great interest in the machine learning community to reduce Deep Neural Network (DNN) model sizes. Decreasing the memory and compute requirements expands the range of resource-constrained mobile applications where DNNs can be deployed. By far, the most popular method of compressing model size is uniform quantization. In this work, we illustrate how quantization performance was fortuitously advanced by Batch Normalization (BatchNorm), a technique originally developed to aid training convergence. This improvement is due to BatchNorm's reshaping of the network's activation distributions. Additionally, due to the limited consensus on why BatchNorm is effective, this work uses concepts from the traditional adaptive filter domain to provide insights into its dynamics and inner workings. First, we show that the convolution weight updates have natural modes whose stability and convergence speed are tied to the eigenvalues of the input autocorrelation matrices. Furthermore, our experiments demonstrate that the speed and stability benefits are distinct effects. At low learning rates, it is BatchNorm's amplification of the smallest eigenvalues that improves convergence speed. In contrast, at high learning rates, it is BatchNorm's suppression of the largest eigenvalues that ensures stability. Next, we prove that in the first training step, when normalization is needed most, BatchNorm satisfies the same optimization as Normalized Least Mean Square (NLMS), while it continues to approximate this condition in subsequent steps. The analyses provided lay the groundwork for gaining further insight into the operation of modern neural network structures using adaptive filter theory. Finally, we highlight contributions made to a real-world application of DNNs in the Smart Hospital space.

ISBN: 9798494462435Subjects--Topical Terms:

588699
Neurons.

Analysis of Quantization and Normalization Effects in Deep Neural Networks.
LDR:02872nmm a2200325 4500 001 2349878
005 20221010063646.5
008 241004s2021 ||||||||||||||||| ||eng d
020 $a 9798494462435
035 $a (MiAaPQ)AAI28812913
035 $a (MiAaPQ)STANFORDgb995rt5179
035 $a AAI28812913
040 $a MiAaPQ $c MiAaPQ
100 1 $a Chai, Elaina Teresa. $3 3689302
245 1 0 $a Analysis of Quantization and Normalization Effects in Deep Neural Networks.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 153 p.
500 $a Source: Dissertations Abstracts International, Volume: 83-05, Section: B.
500 $a Advisor: Murmann, Boris;Mujica, Fernando;Pilanci, Mert.
502 $a Thesis (Ph.D.)--Stanford University, 2021.
506 $a This item must not be sold to any third party vendors.
520 $a There is great interest in the machine learning community to reduce Deep Neural Network (DNN) model sizes. Decreasing the memory and compute requirements expands the range of resource-constrained mobile applications where DNNs can be deployed. By far, the most popular method of compressing model size is uniform quantization. In this work, we illustrate how quantization performance was fortuitously advanced by Batch Normalization (BatchNorm), a technique originally developed to aid training convergence. This improvement is due to BatchNorm's reshaping of the network's activation distributions. Additionally, due to the limited consensus on why BatchNorm is effective, this work uses concepts from the traditional adaptive filter domain to provide insights into its dynamics and inner workings. First, we show that the convolution weight updates have natural modes whose stability and convergence speed are tied to the eigenvalues of the input autocorrelation matrices. Furthermore, our experiments demonstrate that the speed and stability benefits are distinct effects. At low learning rates, it is BatchNorm's amplification of the smallest eigenvalues that improves convergence speed. In contrast, at high learning rates, it is BatchNorm's suppression of the largest eigenvalues that ensures stability. Next, we prove that in the first training step, when normalization is needed most, BatchNorm satisfies the same optimization as Normalized Least Mean Square (NLMS), while it continues to approximate this condition in subsequent steps. The analyses provided lay the groundwork for gaining further insight into the operation of modern neural network structures using adaptive filter theory. Finally, we highlight contributions made to a real-world application of DNNs in the Smart Hospital space.
590 $a School code: 0212.
650 4 $a Neurons. $3 588699
650 4 $a Algorithms. $3 536374
650 4 $a Twins. $3 772649
650 4 $a Neural networks. $3 677449
650 4 $a Artificial intelligence. $3 516317
650 4 $a Computer science. $3 523869
690 $a 0729
690 $a 0800
690 $a 0984
710 2 $a Stanford University. $3 754827
773 0 $t Dissertations Abstracts International $g 83-05B.
790 $a 0212
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28812913