Auto-Encoders for Self-Supervised Learning
Explore the concept of auto-encoders in self-supervised learning, including feature disentanglement, dimension reduction, and their application in downstream tasks. Learn why auto-encoders are essential in neural networks and discover related ideas like de-noising and data reconstruction.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
AUTO-ENCODER Hung-yi Lee
Self-supervised Learning Framework Auto-encoder Self- supervised Self-supervised Learning Pre-train Learned by tasks not requiring label data data Model for Task 2 Model for Task 3 Model for Task 1 Downstream Tasks
Outline Basic Idea of Auto-encoder Feature Disentanglement Discrete Latent Representation More Applications
Unlabeled Images Auto-encoder Sounds familiar? We have seen the same idea in Cycle GAN. (reconstruction) As close as possible high dim vector NN NN Encoder Decoder low dim (bottleneck) Dimension reduction old feature Embedding, Representation, Code New feature for downstream tasks
More Dimension Reduction (not based on deep learning) https://youtu.be/iwh5o_M4BNU https://youtu.be/GBUEjkpoxXc PCA t-SNE
Why Auto-encoder? As close as possible (reconstruction) NN NN Encoder Decoder Find pattern, and then compress 3 x 3 2 dim 3 x 3
Why Auto-encoder? NN Encoder
Auto-encoder is not a new idea Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 313.5786 (2006): 504-507
De-noising Auto-encoder As close as possible Add noises vector NN NN Encoder Decoder The idea sounds familiar? Vincent, Pascal, et al. "Extracting and composing robust features with denoising autoencoders." ICML, 2008.
Reconstruction Review: BERT A de-noising auto-encoder minimize cross entropy softmax Decoder Linear Embedding Encoder BERT Add noise
Outline Basic Idea of Auto-encoder Feature Disentanglement Discrete Latent Representation More Applications
Representation includes information of different aspects Decoder Encoder input image reconstructed object, texture . Decoder Encoder input audio reconstructed content, speaker Decoder Encoder reconstructed input sentence syntax, semantic
Feature Disentangle Decoder Encoder input audio reconstructed content, speaker content information Decoder Encoder input audio reconstructed speaker information https://arxiv.org/abs/1904.05742 https://arxiv.org/abs/1804.02812 https://arxiv.org/abs/1905.05879
In the past Speaker A Speaker B How are you? How are you? Good morning Good morning Today Speaker A Speaker B How are you? Good morning Speakers A and B are talking about completely different things.
Application: Voice Conversion How are you? Decoder Encoder How are you? How are you? Hello Decoder Encoder Hello Hello
Application: Voice Conversion How are you? Encoder How are you? How are you? Decoder Hello Encoder How are you? Hello
Application: Voice Conversion Do you Ph.D.? Encoder Do you Ph.D.? Decoder Japanese Encoder Do you Ph.D.?
Application: Voice Conversion Do you Ph.D.? Encoder Japanese Decoder Japanese Encoder Japanese
Outline Basic Idea of Auto-encoder Feature Disentanglement Discrete Latent Representation More Applications
Discrete Representation Real numbers 0.9 0.1 0.3 0.7 NN NN Encoder Decoder Binary 1 0 0 1 NN NN Encoder Decoder glasses female One-hot 0 0 1 0 NN NN Encoder Decoder class
Discrete Representation https://arxiv.org/abs/1711.00937 Vector Quantized Variational Auto-encoder (VQVAE) vector 3 vector NN NN Encoder Decoder (c.f. attention) Compute similarity vector 1 vector 2 vector 5 Codebook (a set of vectors) Learn from data vector 3 vector 4 The most similar one is the input of decoder. For speech, the codebook represents phonetic information https://arxiv.org/pdf/1901.08810.pdf
https://arxiv.org/abs/1810.02851 Text as Representation Only need a lot of documents to train the model seq2seq2seq auto-encoder Unsupervised Summarization not readable word sequence document document En De Summary? Seq2seq Seq2seq
This is cycle GAN Text as Representation Human written summaries Real or not Let discriminator considers my output as real Di Discriminator word sequence document document Readable En De Summary? Seq2seq Seq2seq
Text as Representation Document: 13 , Summary: Human: 13 Unsupervised: Document: , , Summary: Human: Unsupervised:
Text as Representation Document: 27 , , , 26 60 ,100 Summary: Human: 60 Unsupervised: Document: : , Summary: Human: Unsupervised: :
Tree as Embedding https://arxiv.org/abs/1806.07832 https://arxiv.org/abs/1904.03746
Outline Basic Idea of Auto-encoder Feature Disentanglement Discrete Latent Representation More Applications
Generator As close as possible vector NN NN Encoder Decoder Is it a generator? vector Randomly generate a vector from a distribution NN Image ? Decoder With some modification, we have variational auto-encoder (VAE).
Compression As close as possible vector NN NN Encoder Decoder Low-dim Decompression Compression https://arxiv.org/abs/1708.00838 https://arxiv.org/abs/1703.00395 Lossy
Anomaly Detection Given a set of training data ?1,?2, ,?? Detecting input ? is similar to training data or not. Anomaly Detector ? similar to training data normal outlier, novelty, exceptions Anomaly Detector ? different from training data anomaly
Anomaly Detection Training Data: anomaly Training Data: anomaly Training Data: anomaly
Binary Classification? We only have one class. Training auto-encoder Anomaly Detection Fraud Detection Training data: credit card transactions, ?: fraud or not Ref: https://www.kaggle.com/ntnu-testimon/paysim1/home Ref: https://www.kaggle.com/mlg-ulb/creditcardfraud/home Network Intrusion Detection Training data: connection, ?: attack or not Ref: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html Cancer Detection Training data: normal cells, ?: cancer or not? Ref: https://www.kaggle.com/uciml/breast-cancer-wisconsin-data/home
Approach: Auto-encoder Training Using real human faces to learn an autoencoder As close as possible Encoder Decoder Training data Training data Testing Can be reconstructed Encoder Decoder normal
Approach: Auto-encoder Training Using real human faces to learn an autoencoder As close as possible Encoder Decoder Training data Training data Testing cannot be reconstructed Large reconstruction loss anomaly Encoder Decoder anomaly
More about Anomaly Detection Part 1: https://youtu.be/gDp2LXGnVLQ Part 2: https://youtu.be/cYrNjLxkoXs Part 3: https://youtu.be/ueDlm2FkCnw Part 4: https://youtu.be/XwkHOUPbc0Q Part 5: https://youtu.be/Fh1xFBktRLQ Part 6: https://youtu.be/LmFWzmn2rFY Part 7: https://youtu.be/6W8FqUGYyDo
Concluding Remarks Basic Idea of Auto-encoder Feature Disentanglement Discrete Latent Representation More Applications
