Understanding Gravitational Wave Detection at LIGO & Dataset Overview

Gravity Spy Group 89: Piyoosh, Arash, Michael

About LIGO What s LIGO? The Laser Interferometer Gravitational-Wave Observatory is a large-scale physics experiment and observatory to detect cosmic gravitational waves and to develop gravitational-wave observations as an astronomical tool. Detected the first gravitational wave a few years ago, made headlines Takes advantage of array of ultra high precision sensors to rule out noise, increase SNR against outside noise and internal instruments

More about LIGO Sensors Glitches LIGO does not operate in the realm of Electromagnetism Looks for gravitational oscillations within the fabric of spacetime itself Ultra precise sensors pick up the smallest oscillations from anything that has mass Gravitational waves are hard to detect due to noise from various sources (glitches) Spectrogram of glitches can be learned and detected (supervised) from gravitational waves

Background Information on the Dataset This dataset is an ensemble of anomalies organized into 22 classes of anomaly causing artifacts The dataset is represented as a windowed [in time(s)] spectrogram of sensor signal (800 x 600 before cropping) The window duration is used as a hyper parameter tuning variable The data comes from both locations in Hanford, WA and Livingston, LA. 7966 images (provided split: train: 5587, val: 1200, test: 1179) images are cropped to remove the axis labels/title/etc. Sample data shown below:

Original Paper Original paper tried different models logistic regression: 95% SVM linear: 96.19% kernel: 97.12% custom CNN: 97.67% ensemble model (3 CNNs + 1 SVM w/ hard fusion): 98.21% using original dataset (8583) but we use updated/curated version, slightly smaller (7966)

The Reference Paper George, Daniel & Shen, Hongyu & Huerta, Eliu. (2017). Glitch Classification and Clustering for LIGO with Deep Transfer Learning. In the paper, they applied Transfer Learning using: Inception V3 & V2 (98.84%) ResNet50 (98.84%) VGG16 & 19 (98.21%) (used older version of dataset with 8583 samples, custom 80-20 split)

Tested Image Classification Models Alexnet (2012): the network that started it all (5 Conv, 57.1M par) VGG16 (2014): the sequel (13 Conv, 134M par) ResNet-50 (2016): residual connections (4 res. block sets = 49 Conv, 23.6M par) Inception v3 (2016): factorization, grid size reduction (24.4M par) ResNeXt-50 (2018): cardinality (4 res. block sets = 49 Conv, 23.0M par) MobileNet v2 (2018): depthwise conv, inv residuals (6 bottlenecks = 20 Conv, 2.3M par) EfficientNet-b4 (2019): SOTA, compound scaling (17.6M par)

Procedure Preload network pretrained on ImageNet, resize final FC layer to # of classes, set all weights retrainable Use dataset-provided train/test/val splits Rescale image to network input size (generally 224, except for Inception) Normalize image data to ImageNet mean/std Use Adam optimizer with 0.001 learning rate Learning rate decay of 0.1 every 7 epochs Train for 25 epochs, batch size 32

Results Model Test Accuracy Inception v3 0.973 Resnet-50 0.969 VGG16 0.957 Alexnet 0.964 ResNeXt-50 0.964 MobileNet v2 0.975 EfficientNet-b4 0.975 not much difference in results, VGG worst only needs ~7 epochs to train in reality

Secondary Results Accuracy Plot Confusion Matrix

Additional Experiments with CNN-AutoEncoders AutoEncoder with 6 Convolutional Layers was used to encode and reconstruct the training data This custom network achieve 0.05 Loss on training set, and 0.04 Loss on test set. This network proved to converge very fast with Adam optimizer and Weight Decay (L2 Regularization) 5x5 Kernel sizes were effective in learning the anomalies in the early layers

Encoded Layer Representation in CNN-AutoEncoder Network Architecture: 41 Layer Graph Compressed Weight Representation [Layer #18] 1. 2. 3. 4. 5. 6. DataParallel( (module): ae( (encoder): Sequential( (0): Conv2d(3, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2)) (1): ReLU(inplace=True) (2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (3): Conv2d(128, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2)) (4): ReLU(inplace=True) (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (6): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): Conv2d(64, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2)) (8): ReLU(inplace=True) (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (10): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (11): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (12): ReLU(inplace=True) (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (14): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (15): Conv2d(16, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (16): ReLU(inplace=True) (17): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (18): Conv2d(8, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (19): ReLU(inplace=True) (20): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (21): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (decode): Sequential( (0): Conv2d(4, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU(inplace=True) (2): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (3): Upsample(scale_factor=2.0, mode=bilinear) (4): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (5): ReLU(inplace=True) (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (7): Upsample(scale_factor=2.0, mode=bilinear) (8): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (9): ReLU(inplace=True) (10): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (11): Upsample(scale_factor=2.0, mode=bilinear) (12): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (13): ReLU(inplace=True) (14): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (15): Upsample(scale_factor=2.0, mode=bilinear) (16): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (17): ReLU(inplace=True) (18): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (19): Upsample(scale_factor=2.0, mode=bilinear) (20): Conv2d(128, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) ) ) ) 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51.

References Coughlin, Scott, Updated Gravity Spy Data Set . Zenodo, 01-Nov-2018. https://doi.org/10.5281/zenodo.1476550 Bahaadini S, Noroozi V, Rohani N, Coughlin S, Zevin M, Smith JR et al. Machine learning for Gravity Spy: Glitch classification and dataset. Information Sciences. 2018 May;444:172-186. https://doi.org/10.1016/j.ins.2018.02.068 George, Daniel & Shen, Hongyu & Huerta, Eliu. (2017). Glitch Classification and Clustering for LIGO with Deep Transfer Learning. https://doi.org/10.1103/PhysRevD.97.101501 Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (June 2017), 84 90. https://doi.org/10.1145/3065386 Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014). K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 770-778, doi: 10.1109/CVPR.2016.90. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, "Rethinking the Inception Architecture for Computer Vision," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 2818-2826, doi: 10.1109/CVPR.2016.308. Xie, Saining & Girshick, Ross & Dollar, Piotr & Tu, Z. & He, Kaiming. (2017). Aggregated Residual Transformations for Deep Neural Networks. 5987-5995. 10.1109/CVPR.2017.634. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 4510-4520, doi: 10.1109/CVPR.2018.00474. Tan, Mingxing, and Quoc V. Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." arXiv preprint arXiv:1905.11946 (2019).