Effective Tips for Training with Adaptive Learning Rate
Discover how to navigate a rugged error surface in training processes by implementing adaptive learning rate strategies. Learn the significance of adjusting learning rates based on parameters and explore techniques like RMSProp for dynamic adaptation. Overcome training challenges and optimize performance with tailored learning rate approaches.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Error surface is rugged Tips for training: Adaptive Learning Rate 1
https://docs.google.com/presentation/d/1siUFXARYRpNiMeSRwgFbt7mZVjkMPhR5od09w0Z8xahttps://docs.google.com/presentation/d/1siUFXARYRpNiMeSRwgFbt7mZVjkMPhR5od09w0Z8xa U/edit#slide=id.g3532c09be1_0_382 Training stuck Small Gradient People believe training stuck because the parameters are around a critical point loss norm of gradient 2
Training can be difficult even without critical points. This error surface is convex. Learning rate cannot be one-size-fits-all 100,000 updates ? = 10-2 ? = 10-7 4
Different parameters needs different learning rate Formulation for one parameter: ?+? ?? ? ??? ?=?? ??? ? ?? Smaller Learning Rate ?? |?=?? ?2 ? ? ?+? ?? ? ?? ??? Larger ?? Learning Rate Parameter dependent ?1 5
?? ?+? ?? ? Root Mean Square ?? ??? ?? ? ?2 ? ? ? ?? ? 0= ?? 0?? = ?? ?? ?? ?? ? ? 1 2 ?2+ ?? ?2 ? ?? ? 1= ?? 1?? ?? ?? ?? ? 1 3 ? ?? ? ? ?2+ ?? ?2+ ?? ?2 ?? 2?? 2= ?? ?? ?? ? 1 ? ? ?2 ?= ?? ? + 1 ?? ?+? ?? ? ?? ??? ?? ?=0 6
Root Mean Square ? smaller ?1 larger step ? ? ?1 ? ?1 ? ? ?+? ?? ? ?? ??? ?1 ?? ? ? ?2 ? larger?2 smaller step ? 1 ?2 ?= ?? ? + 1 ?? ? ?2 ?=0 ?2 Used in Adagrad 7
Learning rate adapts dynamically Error Surface can be very complex. Smaller Learning Rate ?2 Larger Learning Rate ?1 8
?? ?+? ?? ? RMSProp ?? ??? ?? ? ?2 0 ? ? ?? 0= ?? 0?? ?? ?? ?? 0 < ? < 1 1 ? 02+ 1 ? ?? ?2 ? ?? ? ?? 1?? 1= ?? ? ?? ?? ? 12+ 1 ? ?? ?2 3 ?? 2 ? ?? 2?? 2= ?? ? ?? ?? ? ? ?+? ?? ? ? 12+ 1 ? ?? ?2 ?? ??? ?= ?? ? ?? ?? 9
??? ? ?? ? ? ?? RMSProp 0 < ? < 1 ? ? ? 12+ 1 ? ?? ?2 ?+? ?? ? ?? ??? ?= ?? ? ?? ?? The recent gradient has larger influence, and the past gradients have less influence. ? small ?? larger step ? increase ?? smaller step ? decrease ?? larger step 10
Original paper: https://arxiv.org/pdf/1412.6980.pdf Adam Adam: RMSProp + Momentum for momentum for RMSprop 11
Without Adaptive Learning Rate ? ? ?+? ?? ? ?? ??? ?? ? 1 ?2 ?= ?? ? + 1 ?? ?=0 12
?? Learning Rate Scheduling ? ? ?+? ?? ? ?? ??? ?? ?? Learning Rate Decay As the training goes, we are closer to the destination, so we reduce the learning rate. ? 13
?? Learning Rate Scheduling ? ? ?+? ?? ? ?? ??? ?? ?? Learning Rate Decay As the training goes, we are closer to the destination, so we reduce the learning rate. ? Warm Up ?? Increase and then decrease? ? 14
Residual Network https://arxiv.org/abs/1512.03385 Transformer https://arxiv.org/abs/1706.03762 15
?? Learning Rate Scheduling ? ? ?+? ?? ? ?? ??? ?? ?? Learning Rate Decay After the training goes, we are close to the destination, so we reduce the learning rate. ? Warm Up ?? Increase and then decrease? At the beginning, the estimate of ?? ?has large variance. ? Please refer to RAdam https://arxiv.org/abs/1908.03265 16
Summary of Optimization (Vanilla) Gradient Descent ?+? ?? ? ??? ? ?? Various Improvements Learning rate scheduling ? ?? ?+? ?? ? ?? ??? Momentum: weighted sum of the previous gradients ?? Consider direction root mean square of the gradients only magnitude 17
To Learn More https://youtu.be/4pUmZ8hXlHM https://youtu.be/e03YKGHXnL8 (in Mandarin) (in Mandarin) 18
Next Time Source of image: https://arxiv.org/abs/1712.09913 Next time Better optimization strategies: If the mountain won't move, build a road around it. Can we change the error surface? Directly move the mountain! 19
