Effective Techniques in Deep Learning for Model Optimization
Explore regularization, data augmentation, and transfer learning concepts to improve model performance in deep learning. Understand the fundamentals of machine learning, minimizing errors, and adjusting model and data sizes. Discover strategies to bring critical values below 1, such as increasing training data and using smaller hypothesis spaces.
Uploaded on | 2 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
CS-EJ3311 Deep Learning with Python Regularization Alexander Jung 19.3.2025 1
What I want to teach you today: What I want to teach you today: basic idea of regularization regularization via data augmentation regularization via transfer learning 2 19.3.2025
What is ML ? What is ML ? informal: learn hypothesis out of a hypothesis space or model that incurs minimum loss when predicting labels of datapoints based on their features training error see Ch. 4.1 of mlbook.cs.aalto.fi 19.3.2025 3
hypothesis h(x) that minimizes ?? ?(?),?(?) label y training error ??=1 3 ( ?? ??)2 validation error ??= ( ?4 ?4)2 3 ?=1 feature x 4 19.3.2025
label y feature x 5 19.3.2025
Data and Model Size Data and Model Size m training set hypospace /model nr. of features n crucial parameter is the ratio d/m d 6 19.3.2025
training error validation error critical value (d/m=1) d / m adjust model and/or data to reach 19.3.2025 7
bring d/m below critical value 1: increase m by using more training data decrease d by using smaller hypothesis space 8 19.3.2025
Data Augmentation 9 19.3.2025
bring d/m below critical value 1: increase m by using more training data decrease d by using smaller hypothesis space 10 19.3.2025
add a bit of noise to features add a bit of noise to features original datapoint label y augmented feature x we have enlarged dataset by factor 3 ! 11 19.3.2025
rotated cat image is still cat image rotated cat image is still cat image 12 19.3.2025
flipped cat image is still cat image flipped cat image is still cat image 13 19.3.2025
shifted cat image is still cat image shifted cat image is still cat image 14 19.3.2025
15 19.3.2025
Transfer Learning 16 19.3.2025
bring d/m below critical value 1: increase m by using more training data decrease d by using smaller hypothesis space 17 19.3.2025
replace original ERM ? 1 ? ?=1 (??,??), min ? with ERM on smaller ? 1 ? ?=1 (??,??), min ? 18 19.3.2025
Prune Network Architecture Prune Network Architecture one hidden layer two hidden layers . three hidden layers 19 19.3.2025
Prune Hypospace by Early Stopping 10 iterations 100 iterations 10000 iterations 20 19.3.2025
reference hypothesis ( pretrained net ) Transfer Learning all possible maps h(.) 21 19.3.2025
Problem I: classify image as shows border collie vs. not Problem II: classify image as shows a dog vs. not ML Problem I is our main interest only little training data ?(1)for Problem I much more labeled data ?(2) for Problem II pre-train a hypothesis on ?(2), fine-tune on ?(1) 22 19.3.2025
?(2) ?(1) learn h by fine-tuning pre-train hypothesis 23 19.3.2025
? 1 ? ?=1 (??,??), + ?( , ) min ? distance to hypothesis which is pre-trained on ?(2) fine tuning on ?(1) 24 19.3.2025
Fine Tuning a Pretrained Net all possible maps h(.) learning rate/step size used during fine tuning determines effective model size 25 19.3.2025
https://www.tensorflow.org/api_docs/python/tf/keras/ applications/vgg16/VGG16 26 19.3.2025
Layer-Wise Fine Tuning fine tune deeper layers freeze input layers cat https://www.quora.com/What-is-the-VGG-neural-network 27 19.3.2025
Feature Extraction frozen input layers perform feature extraction feature extractor or base model head cat https://www.quora.com/What-is-the-VGG-neural-network 28 19.3.2025
https://keras.io/guides/transfer_learning/ 29 19.3.2025
Questions ? 30 19.3.2025
