Encrypted Malware Detection Project at University of South Carolina

Encrypted Malware Detection Alana Mireles, Harikrushna Patel Advisor: Samia Chourieri, Sergio Elizalde, Jorge Crichigno Department of Integrated Information Technology (IIT) University of South Carolina November 14, 2024 1

Agenda Abstract Project Description Background Information Test System Experimentation Results Conclusion 2

Abstract The objective of this project is to address the vulnerabilities posed by the growing trend of encrypted traffic transmission, specifically malware entering networks undetected through TLS channels. Direct client-to-server TLS session can allow malicious content to enter a secured network due to a lack of adequate decryption policy. Traditional firewalls are limited to stateless packet inspection only examining the packet header. NGFW can be configured to decrypt traffic, inspect it, block any threats or re-encrypt and forward the traffic. This project aims at testing the effectiveness of NGFW Decryption Policies at detecting and preventing malware from infiltrating the network through secured TLS channels. 3

Project Description The use of a Palo Alto NGFW splits the traditional client-server TLS session into respective sessions for before and after traffic inspection. The initial session being between the client (e.g., browser) and the NGFW. The following session between the NGFW and the server. Inbound HTTPS traffic is decrypted and inspected accordingly, with malicious content being blocked and logged while safe traffic is forwarded. Decryption Policy Rules along with a Self-Signed Certificate are configured to facilitate the NGFW to act as an SSLForward Proxy. The outcome of this project is to highlight the "Trojan Horse" qualities of HTTPS traffic allowing malware to enter networks and how NGFWs can mitigate these threats from going ignored. 4

Background Information HTTPS is a transport layer protocol which establishes end-to-end encrypted communication between a client and web server. SSL is the security protocol which encrypts, decrypts, and inspects TLS traffic across secured channels. SSL Certificates are a identifiers issued by trusted Certificate Authorities to web servers for the purposes of authentication. A "Trojan Horse" is a type of malware which can enter a secure network by concealing itself in legitimate traffic. A Firewall is a network security tool which inspects and reacts to traffic based on configured security rules. A NGFW provides improved security tools with enhanced packet inspection, intrusion prevention, policy management, etc. An SSL Forward Proxy is essentially a device which acts as the man in the middle responsible for the inspection and forwarding of encrypted traffic in client-to-server connection. 5

Test System In terms of this project, the lab infrastructure included a DMZ server, a client end device, a Palo Alto NGFW, and a router to the external network. The internal network represents the trusted network, where client devices are located. The DMZ is a layer of separation for the servers that may need to be accessed from both inside and outside the network. The firewall controls traffic between these zones ensuring security policies are enforced across the network. The external network represents the untrusted network, where the router is located that connects to the internet. System Topology 6

Experimentation The Decrypt Policy Rule is configured to decrypt any URL Category/Service traffic between the connections from the Users_Net zone to the Extranet & Internet zones. Following the decryption and inspection of traffic, it is then re-encrypted and forwarded via a new TLS session. The No-Decrypt Policy Rule is created to exempt the traffic of the URL categories financial services, government, and shopping from being inspected due to the factor that it may contain PII. The No-Decrypt rule is prioritized first to filter out excluded traffic containing PII prior to inspection. Decrypt & No-Decrypt Policies 7

Experimentation The new TLS session created after the decryption policies will require the NGFW issue a new SSL Certificate. A Self-Signed Certificate is generated on the NGFW for connections to web servers with certificates signed by trusted Certificate Authorities. The newly generated certificate is imported to the client browser's certificate manager in order to recognize the NGFW as a Certificate Authority. Self-Signed Trust Certificate 8

Experimentation Imported Certificate Manager 9

Results The certificate in the next slide demonstrates that the traffic has been decrypted, inspected, and then re-encrypted as it initially wasn't trusted by the client browser. We can determine this by looking at the issuer name of 192.168.1.1, the name of the firewall on the client browser. After the decryption policy had taken effect, it was observed in the threat log that the traffic was a threat and had been decrypted. With this, we can verify that the decryption policies and trust certificates have been correctly configured to decrypt and re-encrypt the TLS sessions. 10

Results Certificate Trust in Effect 11

Results Threat Log 12

Results We were also able to exclude gov, shopping, and finance URLs from decryption as to avoid exposing Personally Identifiable Information (PII). With this, we were able to verify that the no-decryption policy was in effect when viewing the certificate on the Texas page that displays the issuer name coming from Amazon instead of 192.168.1.1 or similar. 13

Conclusion As demonstrated, we have successfully configured trust certificates and decryption policies to detect potential malware within encrypted traffic that could otherwise reach client devices undetected. Testing confirmed that the decryption policies are functioning correctly, as indicated by the issuer name "192.168.1.1" on certificates from certain sites. This shows that the firewall is actively performing the appropriate inspections. In conclusion, the firewall s decryption, inspection, and re-encryption policies are effectively configured, enabling threat detection while maintaining privacy on sensitive sites. 14