東華大學圖書館 |

語系: 繁體中文

說明(常見問題)

回圖書館首頁

手機版館藏查詢

登入

回首頁

切換: 標籤 | MARC模式 | ISBD

FindBook

Google Book

Amazon

博客來

Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent./
作者:	Backstrom, Karl.
面頁冊數:	1 online resource (161 pages)
附註:	Source: Dissertations Abstracts International, Volume: 85-03, Section: B.
Contained By:	Dissertations Abstracts International85-03B.
標題:	Software. -
電子資源:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30565531click for full text (PQDT)
ISBN:	9798380263542

Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent.
Backstrom, Karl.

Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent. - 1 online resource (161 pages)

Source: Dissertations Abstracts International, Volume: 85-03, Section: B.

Thesis (Ph.D.)--Chalmers Tekniska Hogskola (Sweden), 2023.

Includes bibliographical references

Accelerated digitalization and sensor deployment in society in recent years poses critical challenges for associated data processing and analysis infrastructure to scale, and the field of big data, targeting methods for storing, processing, and revealing patterns in huge data sets, has surged. Artificial Intelligence (AI) models are used diligently in standard Big Data pipelines due to their tremendous success across various data analysis tasks, however exponential growth in Volume, Variety and Velocity of Big Data (known as its three V's) in recent years require associated complexity in the AI models that analyze it, as well as the Machine Learning (ML) processes required to train them. In order to cope, parallelism in ML is standard nowadays, with the aim to better utilize contemporary computing infrastructure, whether it being sharedmemory multi-core CPUs, or vast connected networks of IoT devices engaging in Federated Learning (FL).Stochastic Gradient Descent (SGD) serves as the backbone of many of the most popular ML methods, including in particular Deep Learning. However, SGD has inherently sequential semantics, and is not trivially parallelizable without imposing strict synchronization, with associated bottlenecks. Asynchronous SGD (AsyncSGD), which relaxes the original semantics, has gained significant interest in recent years due to promising results that show speedup in certain contexts. However, the relaxed semantics that asynchrony entails give rise to fundamental questions regarding AsyncSGD, relating particularly to its stabilityand convergence rate in practical applications.This thesis explores vital knowledge gaps of AsyncSGD, and contributes in particular to: Theoretical frameworks - Formalization of several key notions related to the impact of asynchrony on the convergence, guiding future development of AsyncSGD implementations; Analytical results - Asymptotic convergence bounds under realistic assumptions. Moreover, several technical solutions are proposed, targeting in particular: Stability - Reducing the number of non-converging executions and the associated wasted energy; Speedup - Improving convergence time and reliability with instance-based adaptiveness; Elasticity- Resource-efficiency by avoiding over-parallelism, and thereby improving stability and saving computing resources. The proposed methods are evaluated on several standard DL benchmarking applications and compared to relevant baselines from previous literature. Key results include: (i) persistent speedup compared to baselines, (ii) increased stability and reduced risk for non-converging executions, (iii) reduction in the overall memory footprint (up to 17%), as well as the consumed computing resources (up to 67%).In addition, along with this thesis, an open-source implementation is published, that connects high-level ML operations with asynchronous implementations with fine-grained memory operations, leveraging future research for efficient adaptation of AsyncSGDfor practical applications.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2023

Mode of access: World Wide Web

ISBN: 9798380263542Subjects--Topical Terms:

619355
Software.
Index Terms--Genre/Form:

542853
Electronic books.

Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent.
LDR:04273nmm a2200325K 4500 001 2363985
005 20231127094752.5
006 m o d
007 cr mn ---uuuuu
008 241011s2023 xx obm 000 0 eng d
020 $a 9798380263542
035 $a (MiAaPQ)AAI30565531
035 $a (MiAaPQ)Chalmers_SE535694
035 $a AAI30565531
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Backstrom, Karl. $3 3704769
245 1 0 $a Adaptiveness, Asynchrony, and Resource Efficiency in Parallel Stochastic Gradient Descent.
264 0 $c 2023
300 $a 1 online resource (161 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-03, Section: B.
500 $a Advisor: Papatriantafilou, Marina.
502 $a Thesis (Ph.D.)--Chalmers Tekniska Hogskola (Sweden), 2023.
504 $a Includes bibliographical references
520 $a Accelerated digitalization and sensor deployment in society in recent years poses critical challenges for associated data processing and analysis infrastructure to scale, and the field of big data, targeting methods for storing, processing, and revealing patterns in huge data sets, has surged. Artificial Intelligence (AI) models are used diligently in standard Big Data pipelines due to their tremendous success across various data analysis tasks, however exponential growth in Volume, Variety and Velocity of Big Data (known as its three V's) in recent years require associated complexity in the AI models that analyze it, as well as the Machine Learning (ML) processes required to train them. In order to cope, parallelism in ML is standard nowadays, with the aim to better utilize contemporary computing infrastructure, whether it being sharedmemory multi-core CPUs, or vast connected networks of IoT devices engaging in Federated Learning (FL).Stochastic Gradient Descent (SGD) serves as the backbone of many of the most popular ML methods, including in particular Deep Learning. However, SGD has inherently sequential semantics, and is not trivially parallelizable without imposing strict synchronization, with associated bottlenecks. Asynchronous SGD (AsyncSGD), which relaxes the original semantics, has gained significant interest in recent years due to promising results that show speedup in certain contexts. However, the relaxed semantics that asynchrony entails give rise to fundamental questions regarding AsyncSGD, relating particularly to its stabilityand convergence rate in practical applications.This thesis explores vital knowledge gaps of AsyncSGD, and contributes in particular to: Theoretical frameworks - Formalization of several key notions related to the impact of asynchrony on the convergence, guiding future development of AsyncSGD implementations; Analytical results - Asymptotic convergence bounds under realistic assumptions. Moreover, several technical solutions are proposed, targeting in particular: Stability - Reducing the number of non-converging executions and the associated wasted energy; Speedup - Improving convergence time and reliability with instance-based adaptiveness; Elasticity- Resource-efficiency by avoiding over-parallelism, and thereby improving stability and saving computing resources. The proposed methods are evaluated on several standard DL benchmarking applications and compared to relevant baselines from previous literature. Key results include: (i) persistent speedup compared to baselines, (ii) increased stability and reduced risk for non-converging executions, (iii) reduction in the overall memory footprint (up to 17%), as well as the consumed computing resources (up to 67%).In addition, along with this thesis, an open-source implementation is published, that connects high-level ML operations with asynchronous implementations with fine-grained memory operations, leveraging future research for efficient adaptation of AsyncSGDfor practical applications.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2023
538 $a Mode of access: World Wide Web
650 4 $a Software. $2 gtt. $3 619355
650 4 $a Privacy. $3 528582
655 7 $a Electronic books. $2 lcsh $3 542853
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 783688
710 2 $a Chalmers Tekniska Hogskola (Sweden). $3 1913472
773 0 $t Dissertations Abstracts International $g 85-03B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30565531 $z click for full text (PQDT)