Recurrent Neural Networks: Applications and Training
Explore the world of Recurrent Neural Networks (RNN) through examples and explanations. Learn about the advantages of RNN, its application in handling sequence data, and the process of training RNN models. Discover how RNNs can be unfolded in time for efficient learning and understand the architecture of a Vanilla RNN cell. Dive into the challenges like gradient vanishing while training RNNs and explore various input-output scenarios.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Introduction to Recurrent Neural Networks (RNN) J.-S. Roger Jang ( ) jang@mirlab.org http://mirlab.org/jang MIR Lab, CSIE Dept. National Taiwan University 2025/4/12
NN vs. RNN Two types of tasks in machine learning Tasks of fixed-dimension input MNIST database (28x28 inputs), Iris dataset (4 inputs), Tasks of variable-dimension input Text, speech, singing So we have two types of classifiers Static classifiers with fixed-dimension input KNNC, QC, NBC, SVM, NN, random forest, Sequence classifiers for variable-dimension input HMM, RNN, CRF 2/33
Recurrent Neural Networks (RNN) RNN NN that uses values from the previous step Or equivalently, NN that has memory Advantage of RNN Good for sequence input Example Compute the cumulated sum of a sequence Input=[1 2 3], output=6 Input=[1 3 4 5 2 4 3], output=22 Output 1 if the number of 1s is odd Input= 100011, output=1 Input=1000010101, output=0 4/33
Unfold in Time for RNN RNNs can be transformed into a feedforward static mapping via unfold in time Training is based on BPTT (back propagation through time) over the unfolded network Unfolded network! 5/33
Vanilla RNN Cell and Its Unfolded Network The vanilla RNN cell Unfolded network Feedforward, ready for BP (backpropagation) y1 W xt y2 y3 ht ht-1 h1 h2 h3 x t = + tanh h W b t h t-1 Weights are shared over time! x1 h0 x2 h1 x3 h2 6/33
BP for Unfolded Network Unfolded network Feedforward BP as usual! h0 x1 h1 x2 h2 x3 h3 Gradient vanishing problem Especially severe in RNN y2 y3 y1 y y y y y 1 ( ) ( y ) ( y ) = = 3 y 1 y 1 y 1 y 2 3 3 2 2 1 1 y y y y 8 1 3 2 1 7/33
Input-Output Scenarios Single - Single Feed-forward Network Single - Multiple Image Captioning Multiple - Single Sentiment Classification Multiple - Multiple Translation Image Captioning 8/33
Long Short-Term Memory (LSTM) Add a memory cell that is not subject to matrix multiplication or squishing, thereby avoiding gradient vanishing xt ht-1 ht ct-1 ct S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Computation 9 (8), pp. 1735 1780, 1997
The LSTM Cell Wg xt Cell ct ht= tanh?? ?? ??= tanh?? ? 1 ht-1 ??= ?? 1+ ?? * Dashed line indicates time-lag
The LSTM Cell Wg xt Cell ct ht ?? ??= tanh?? ? 1 ht-1
The LSTM Cell xt ht-1 Wi Input Gate it ?? ??= ? ?? + ?? ? 1 Wg xt Cell . ct ht ?? ??= tanh?? ? 1 ht-1 ??= ?? 1+ ?? ??
The LSTM Cell xt ht-1 xt ht-1 Wi Wo Input Gate Output Gate it ot ?? ?? ??= ? ?? + ?? ??= ? ?? + ?? ? 1 ? 1 Wg xt Cell . . ct ht ?= ?? tanh?? ?? ??= tanh?? ? 1 ht-1 ??= ?? 1+ ?? ??
The LSTM Cell xt ht-1 xt ht-1 Wi Wo Input Gate Output Gate it ot ?? ?? ??= ? ?? + ?? ??= ? ?? + ?? ? 1 ? 1 Wg xt Cell . . ct ht ?= ?? tanh?? ?? ??= tanh?? ? 1 ht-1 ??= ?? ?? 1+ ?? ?? Forget Gate ?? ??= ? ?? + ?? ft ? 1 Wf xt ht-1
Deep RNN RNN with multiple hidden layers y1 y4 y2 y3 y5 y6 x1 x4 x2 x3 x5 x6
Bidirectional RNNs RNNs can process the input sequence in forward and in the reverse direction y1 y2 y3 y4 y5 y6 x1 x4 x2 x3 x5 x6
RNN for Singing Transcription Basic idea xi Feature vector of frame i oi Output vector of frame i = [onset prob, offset prob] 1 o 5 o 3 o n o 2 o 4 o 3 x 5 x 1 x x x x 2 4 n Questions How to reinforce the correct order of onset/offset? How to determine the note pitch (pitch between onset and offset)? 17/33
References Some course material taken from the following slides http://slazebni.cs.illinois.edu/spring17/lec02_rnn.pptx by Arun Mallya http://fall97.class.vision/slides/5.pptx https://cs.uwaterloo.ca/~mli/Deep-Learning-2017- Lecture6RNN.ppt Videos https://www.youtube.com/watch?v=xCGidAeyS4M by https://www.youtube.com/watch?v=rTqmWlnwz_0 by https://www.youtube.com/watch?v=lycKqccytfU 18/33
