Defense Against Adversarial Images using Web-Scale Nearest-Neighbor Search
This study explores a method to defend against adversarial images by approximating their projection onto the image manifold through nearest-neighbor search. The approach involves finding the nearest neighbors in a web-scale image database to classify and mitigate the impact of adversarial perturbations. Various techniques, such as measuring Euclidean distances in feature space and identifying key features, are employed to enhance the accuracy of this defense mechanism.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Defense Against Adversarial Images using Web-Scale Nearest-Neighbor Search 2025/2/16 Yujia Liu 1
Hypothesis There exists no physical process that could have produced the adversarial image. + Fact Many adversarial perturbations may be considered as transformations that take a sample from the image manifold and move it away from that manifold. 2025/2/16 Yujia Liu 2
Method off-manifold adversarial images approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database classify the projection of the adversarial example 2025/2/16 Yujia Liu 3
Method approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database Feature of the adversarial sample Features of images from the database measuring Euclidean distances in feature space Find K nearest neighbors from the database 2025/2/16 Yujia Liu 4
Method approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database Feature of the adversarial sample Features of images from the database measuring Euclidean distances in feature space Find K nearest neighbors from the database 2025/2/16 2025/2/16 Yujia Liu Yujia Liu 5 5
Method approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database Feature of the adversarial sample Features of images from the database measuring Euclidean distances in feature space Find K nearest neighbors from the database 2025/2/16 2025/2/16 2025/2/16 Yujia Liu Yujia Liu Yujia Liu 6 6 6
Method approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database Features (1) conv_5 5_1 1:pre-ReLU activations from the conv_5_1 layer of a ResNet-50 trained on ImageNet-1K + reduce these features to 256 dimensions using a spatial average pooling followed by PCA (Principal Component Analysis) (2) conv_5 5_1 1- -RMAC RMAC: conv_5_1 features from a ResNet-50 followed by R-MAC pooling , bit quantization, and dimensionality reduction [1] [1] G. Tolias, R. Sicre, and H. J egou. Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879, 2015. 5 2025/2/16 Yujia Liu 7
Method approximate the projection of an adversarial example onto the image manifold by the finding nearest neighbors in the image database Database (1)IG Database: IG-N-All and IG-N-targeted ( N = 1/50 billion ) ( ECCV 2018) (2) YFCC100M: 100 million Flickr images with associated meta-data (2015) (3) IN-1.3M: training spilt of ImageNet 2025/2/16 Yujia Liu 8
Method classify the projection of the adversarial example 2025/2/16 Yujia Liu 9
Method classify the projection of the adversarial example Choose top K nearest classes 1? (1) Uniform Weighting (UW): w = (2) CBW-E(ntropy): ?:softmax vector ?: number of classes (3) CBW-D(iversity): ?:the sorted version of the softmax vector (descending order) ? = 20 ? = 3 2025/2/16 Yujia Liu 10
Experiment Setup: K =50 Dataset: ImageNet Attack method: FGSM Classifier: ResNet-50 Black-box generate model: ResNet-18 Table 1: ImageNet classification accuracy of ResNet-50. 2025/2/16 Yujia Liu 11
Experiment _ K Setup: Dataset: ImageNet Attack method: FGSM Feature: conv_5_1 Classifier: ResNet-50 Weight: CBW-D 2025/2/16 Yujia Liu 12
Experiment _ Features Setup: Dataset: ImageNet Attack method: FGSM Classifier: ResNet-50 Searching dataset: IG-1B-targeted 2025/2/16 Yujia Liu 13
Experiment _ Compare with STOA Defenses 2025/2/16 Yujia Liu 14
Conclusion & Comment + demonstrate the feasibility of web-scale nearest-neighbor search as a defense mechanism + provide a new avenue for defending methods - computational cost is large, the results benefit from their powerful hardware 2025/2/16 Yujia Liu 15
