Advantages of Deep Learning in Neural Networks

why deep learning n.w
1 / 24
Embed
Share

Explore the importance of deep learning, why hidden layers are crucial, and the benefits of piecewise linear functions. Discover the power of deep networks over fat networks and how the depth of a network affects word error rates in conversational speech transcription.

  • Deep Learning
  • Neural Networks
  • Advantages
  • Deep Networks
  • Word Error Rate
rayaanacharya
sharpton_m Follow

Uploaded on | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. Why Deep Learning? Hung-yi Lee

  2. smaller larger ? ?????,???? ? ?????,???? small large ? ???,???? large ? ???,???? small Still small loss ???= ???min ? ,???? Small (fewer candidates)

  3. Review: Why Hidden Layer?

  4. Piecewise Linear https://youtu.be/bHcJCp2Fyxs 0 1 We can have good approximation with sufficient pieces.

  5. piecewise linear = constant + sum of a set of 0 1

  6. sum of a set of = constant + Piecewise linear How to represent this function? Hard Sigmoid ?1 Sigmoid Function 1 ? = ? 1 + ? ?+??1 = ? ??????? ? + ??1 ?1

  7. sum of a set of = constant + Piecewise linear ?1 ?1 ?11 + ?1 ?1 ?1 ?12 1 ?13 ?2 ?2 ? ?2 + ?2 + ? ?3 1 1 ?3 ?3 ?3 + 1

  8. Hard Sigmoid ReLU How to represent this function? ?1 Rectified Linear Unit (ReLU) ? ??? 0,? + ??1 ?1 ? ??? 0,? + ? ?1

  9. sum of a set of = constant + Piecewise linear + = ?1 ?1 ?11 + ?1 ?1 ?1 ?12 1 ?13 ?2 ?2 ? ?2 + ?2 + ? ?3 1 1 ?3 ?3 ?3 + 1 Why we want Deep network, not Fat network?

  10. Deeper is Better? Word Error Rate (%) Word Error Rate (%) larger Layer X Size Layer X Size 1 X 2k 24.2 ? ?????,???? 2 X 2k 3 X 2k 20.4 18.4 large 4 X 2k 17.8 ? ???,???? 5 X 2k 17.2 1 X 3772 small 22.5 7 X 2k 17.1 1 X 4634 22.6 1 X 16k 22.1 Seide Frank, Gang Li, and Dong Yu. "Conversational Speech Transcription Using Context-Dependent Deep Neural Networks." Interspeech. 2011.

  11. Fat + Short v.s. Thin + Tall The same number of parameters Which one is better? x 1x 2x x 1x 2x N N Deep Shallow

  12. Fat + Short v.s. Thin + Tall Word Error Rate (%) Word Error Rate (%) Layer X Size Layer X Size 1 X 2k 24.2 2 X 2k 3 X 2k 20.4 18.4 Why? 4 X 2k 17.8 5 X 2k 17.2 1 X 3772 22.5 7 X 2k 17.1 1 X 4634 22.6 1 X 16k 22.1 Seide Frank, Gang Li, and Dong Yu. "Conversational Speech Transcription Using Context-Dependent Deep Neural Networks." Interspeech. 2011.

  13. Why we need deep?

  14. Why we need deep? Yes, one hidden layer can represent any function. However, using deep structure is more effective. Shallow More parameters > Deep

  15. Analogy Logic Circuits E.g., parity check For input sequence with d bits, 1 0 1 0 1 (even) Circuit Two-layer circuit need O(2d) gates. 0 0 0 1 0 (odd) Circuit XNOR 1 0 1 0 0 0 1 With multiple layers, we need only O(d) gates.

  16. Analogy Programming Don t put everything in your main function. http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

  17. More Analogy

  19. ? 0.5 1 + 2 0.5 1 ? ?1 + ? + 0.5 2 + 1 0.5 1 0 ? 1 ?1

  20. ?1 ? 0.5 0.5 1 2 + + 2 0.5 2 1 1 ? ?2 + 2 ?1 ? + 0.5 +0.5 2 + + 1 2 0.5 1 1 0 0 0 22pieces ? ? ?1 1 1 1 ?1 ?2 ?2

  21. ?2 ?1 0.5 ? 0.5 0.5 1 2 + + + 2 0.5 2 1 1 1 ? ?3 ? + 0.52 ?2+ 0.5 ?1+ 0.5 2 + + + 1 2 0.5 1 1 1 0 0 0 23pieces ?2 ? ? 1 1 1 ?3 ?2 ?3

  22. Deep 2 neurons 2?pieces .. ? ? 0 ? layers smaller (2? neurons) ? Shallow 2?neurons 1 ? larger ? ?

  23. Thinks more Deep networks outperforms shallow ones when the required functions are complex and regular. Image, speech, etc. have this characteristics. Deep is exponentially better than shallow even when ? = ?2. https://youtu.be/FN8jclCrqY0 https://youtu.be/qpuLxXrHQB4

  24. Still small loss ???= ???min ? ,???? Small (fewer candidates)

Related


No related presentations.

More Related Content