How Web Protocol Weaknesses Enable Layer 7 DoS Attacks
What Are Layer 7 DoS Attacks?
Layer 7 (Application Layer) DoS (Denial of Service) attacks target the top of the OSI 7 model by overwhelming server resources, such as CPU, memory, or disk I/O, through seemingly legitimate application requests (e.g., HTTP or HTTPS). Unlike traditional DoS attacks that flood networks at Layer 3 or 4, Layer 7 attacks are low-bandwidth, stealthy, and closely mimic legitimate user behavior, making them difficult to detect and mitigate. These attacks exploit application-specific behaviors at the OSI model application layer, by sending requests that appear valid but trigger resource-intensive operations. As a result, they exhaust server processing power and often bypass firewalls and intrusion detection/prevention systems as the traffic resembles real users. Traditional security tools—which typically monitor traffic volume rather than request complexity—often fail to identify and block such threats.
This article delves into the inherent vulnerabilities in the design of web protocols that indirectly expose applications to Layer 7 DoS attacks, regardless of the use of modern transport encryption mechanisms such as WTLS, DTLS, TLS 1.2/1.3, and GQUIC. The blog will also methodically dissect the specific weaknesses of the ICP and other protocols, using live demonstrations from an accompanying webinar. It will show how these weaknesses can be weaponized to bypass security defenses. Additionally, the analysis will also explore vulnerabilities within the handshake processes of Layer 7 protocols, including DTLS, GQUIC, TLS 1.2/1.3, and WTLS.
What Is Internet Cache Protocol (ICP)?
The Internet Cache Protocol (ICP) was designed to facilitate communication between web caches to locate specific web objects in neighboring caches to reduce latency and server load. It uses a lightweight messaging format to communicate within a mesh of caches, sending queries to determine whether the requested content is already stored locally. This approach helps optimize bandwidth usage and improves efficiency in content delivery systems such as Squid. Although ICP played a vital role in early caching strategies, its relevance has diminished with the advent of more advanced protocols, such as HTTP/1.1. Nevertheless, it remains in limited use within some legacy caching environments.
Examples of security risks associated with ICP include:
- Bypassing HTTP Controls/Standard HTTP Processing: The ICP_OP_HIT_OBJ operation bypasses crucial HTTP controls such as authentication and age validation, potentially allowing unauthorized access to sensitive information, including private user data and restricted content. This can ultimately result in data leaks.
- User Datagram Protocol (UDP) Fragmentation: ICP_OP_HIT_OBJ messages can exceed the path’s maximum transmission unit (MTU), causing User Datagram Protocol (UDP) fragmentation. This fragmentation introduces server overhead due to the need for packet reassembly and creates an opportunity for DoS attacks. Attackers can exploit this by flooding the server with fragmented packets, consuming system resources, and disrupting legitimate traffic—often with minimal bandwidth requirements (Sołtysik, 2025b).
- Cache Poisoning: Because ICP_OP_HIT_OBJ bypasses standard HTTP processing, attackers can inject malicious content directly into the cache. By sending crafted HTTP responses or manipulating peer cache behavior, an attacker may cause a misconfigured cache to store and serve malicious versions of web resources, such as JavaScript files or login pages, which can lead to malware distribution, credential theft, or redirection to malicious domains.
“In practice, I believe HTTP/1.1 made ICP redundant. I doubt there’s a web cache on the other end actually receiving these requests since web caching and ICP are historical relics. I hold that ICP was a cute idea that served a purpose pre-HTTP/1.1, but which has little value today.” – Peter Danzig, creator of ICP.
ICP remains supported in legacy systems, particularly in older Squid deployments and hierarchical cache networks, but its use is largely discouraged in modern environments. While Squid still includes ICP for backward compatibility, most contemporary configurations favor more secure and efficient protocols such as HTTP/1.1, HTTP/2, or ICAP. ICP is considered obsolete due to its security vulnerabilities and limited performance benefits, and deactivating it is recommended in current web caching systems. Although it may persist in a few niche use cases, ICP has been largely replaced as modern networks prioritize more robust and scalable alternatives.
Example 1: Before launching various DoS attacks simultaneously, a round-trip time (RTT) baseline was established using ICMP. Two minutes later, ICP was used to measure RTT for comparison.
Example 2: ICP was used for continuous RTT measurement. Each packet provided data to the attacker.
What Is Wireless Transport Layer Security (WTLS)?
Wireless Transport Layer Security (WTLS) is a security protocol developed for the Wireless Application Protocol (WAP), designed to secure communications over mobile networks, which typically experience higher latency and limited bandwidth. Based on TLS 1.0, WTLS provides encryption and data integrity specifically adapted to the constraints of wireless environments. WTLS cipher suites specify bulk encryption algorithms (e.g., RC5, DES, or null) and message authentication code (MAC) algorithms like SHA-1. Symmetric encryption ensures confidentiality, while MACs provide data integrity by detecting message alterations during transit.
The SSL ChangeCipherSpec phase is a critical part of the handshake process in which the client and server finalize their agreement on encryption algorithms and transition from plaintext to encrypted communication. After negotiating cryptographic parameters and exchanging certificates, both parties send a ChangeCipherSpec message, which is immediately followed by a Finished message encrypted with the selected cipher. This step secures all subsequent data transmission. However, it also presents a potential attack surface: an adversary can exploit this phase by initiating multiple ChangeCipherSpec requests, forcing the server to perform repeated, resource-intensive operations like key generation and encryption. Such abuse can lead to DoS conditions and exhaust the server CPU and memory resources.
Example 3: WTLS ChangeCipherSpec messages attempted to exhaust server resources.
What Is Datagram Transport Layer Security (DTLS)?
Datagram Transport Layer Security (DTLS), as part of the Layer 7 protocols, provides TLS-equivalent security for applications that use UDP instead of Transmission Control Protocol (TCP). Because UDP lacks guarantees for message delivery, ordering, and error correction, DTLS includes mechanisms like fragmentation and retransmission to handle these challenges. It ensures core security features such as encryption, authentication, and data integrity for datagram-based communication. DTLS is widely used in low-latency, security-sensitive scenarios like Voice over IP (VoIP), video conferencing, virtual private networks (VPNs), and Internet of Things (IoT) devices. One of its key advantages is preserving UDP’s low-latency and unordered delivery characteristics while delivering the robust security of Transport Layer Security (TLS).
Despite its robust security features, DTLS remains vulnerable to certain types of DoS attacks. Two primary examples include:
- Resource Exhaustion: Attackers flood the server with numerous ClientHello (CHLO) messages to initiate handshakes. The server must allocate CPU and memory resources and perform computationally intensive tasks, such as key exchange and certificate validation, for each request. Doing so can quickly drain server resources, leading to performance degradation or even system crashes.
- Amplification Attack: In this case, attackers spoof the IP address of a victim and send forged connection requests to the server. The server responds with large Certificate messages directed to the victim’s IP address, overwhelming it with unsolicited traffic and effectively causing a DoS.
The Stateless Cookie Mechanism
In DTLS, the stateless cookie mechanism—inspired by protocols such as Photuris and the Internet Key Exchange Version 2 (IKEv2)—mitigates resource exhaustion attacks. When a client initiates a handshake by sending a CHLO message, the DTLS server responds with a HelloVerifyRequest message that includes a stateless cookie. The client must then resend the CHLO along with the cookie, and only after validating the cookie does the server proceed with the full handshake. This approach ensures that the server does not allocate significant resources or perform cryptographic operations until the client’s IP address is verified, thereby reducing exposure to resource exhaustion. It also protects against IP spoofing, as attackers who cannot receive the cookie cannot complete the handshake, limiting the potential for amplification attacks. However, this mechanism has its limitations. While effective against spoofed IP-based attacks, it does not prevent abuse by attackers using valid IP addresses, who can legitimately receive and return the cookie. In such cases, adversaries may still exhaust server resources by initiating a high volume of valid handshake requests, bypassing the stateless protection and potentially degrading service availability.
Example 4: DTLS handshake abuse occurred when dozens of CHLO messages were sent from the same IP—an address linked to the known scanning service Censys.
What Is QUIC (Quick UDP Internet Connections)?
QUIC (Quick UDP Internet Connections) is a modern transport protocol initially developed by Google as GQUIC and later standardized by the IETF (Internet Engineering Task Force). Designed to run over UDP, QUIC enables faster connection establishment than traditional TCP. It now forms the foundation of HTTP/3, replacing the legacy TCP+TLS stack to significantly enhance web performance. QUIC integrates transport and encryption into a single protocol, with TLS 1.3 built-in by design. This tight integration reduces handshake latency by requiring fewer round trips to establish a secure connection. Additionally, QUIC supports multiplexing, allowing multiple independent data streams within a single connection, which eliminates the head-of-line (HoL) blocking issue commonly associated with TCP, further improving speed and efficiency.
QUIC vs. TLS
QUIC is a modern transport protocol that runs over UDP, offering faster connections than traditional TLS over TCP. It integrates TLS 1.3 directly into the transport layer, supports 0-RTT reconnections, handles packet loss efficiently, and enables seamless connection migration across networks. These features make QUIC faster, more efficient, and more resilient, leading to its widespread adoption by platforms like Google, YouTube, and Cloudflare. It now serves as the foundation of HTTP/3, replacing the older TCP+TLS stack.
GQUIC vs. IETF QUIC
GQUIC was an experimental version of the QUIC protocol that used a proprietary CHLO/SHLO (ClientHello/ServerHello) handshake. IETF QUIC, the standardized version, replaces this with a more secure and interoperable TLS 1.3-based handshake. IETF QUIC provides enhanced encryption, improved extensibility, and stronger protection against spoofing and replay attacks. It is now widely adopted for HTTP/3, with support from major browsers like Chrome and Firefox, as well as services such as Cloudflare and most CDNs. GQUIC has been deprecated and is no longer recommended for use in modern infrastructure.
CHLO Message as an Attack Vector
CHLO is the initial handshake message in GQUIC sent by the client to initiate a session. It contains version information, public keys, and connection parameters. While not inherently vulnerable, CHLO messages can be exploited in DoS attacks, where attackers flood servers with repeated requests. These attacks often reuse identical Connection Identifiers (CIDs), causing the server to allocate resources for each seemingly new connection, leading to CPU and memory exhaustion before the handshake completes. Additionally, GQUIC’s reliance on UDP and its unauthenticated, encrypted handshake messages make it especially vulnerable to reflection attacks, where spoofed CHLO requests trigger the server to send amplified responses to a victim’s IP address, overwhelming their network.
Why GQUIC CHLO Flooding Is Effective
In GQUIC, CHLO flooding is an effective attack vector due to UDP’s stateless nature, which allows attackers to spoof CHLO messages without completing a full connection. Each CHLO requires CPU-intensive parsing by the server, and early versions of GQUIC lacked effective stateful validation. This makes it easy to exhaust server resources. Additionally, GQUIC is vulnerable to reflection-based DoS attacks, where spoofed CHLOs trigger amplified responses directed at a victim, enabled by the absence of early authentication. In contrast, IETF QUIC addresses these weaknesses using stateless retry tokens and integrated TLS 1.3 encryption, making it resilient to such CHLO-based attacks.
Example 5: CHLO flood attack using GQUIC: The source IP generated an overwhelming number of initial handshake packets directed at the same target IP, with the same CID, and the payload size was consistent.
IETF QUIC Still Vulnerable to Layer 7 DoS Attacks
Despite its advancements, IETF QUIC remains vulnerable to Layer 7 DoS attacks. Running TLS 1.3 over UDP, involves resource-intensive operations, such as Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) key exchange, certificate verification, and session state management. While features like 0-RTT and 1-RTT reduce latency, the initial handshake still imposes a significant computational load on servers, especially under high-volume conditions, such as when processing 100,000 handshakes per second.
The Quiche tool, available on GitHub, provides a high-level API for sending and receiving HTTP requests and responses over the QUIC transport protocol.
Mitigating Handshake Flooding in IETF QUIC: Crypto Challenges via Retry Tokens
Even though IETF QUIC is designed with strong security and performance improvements, research from December 2024 shows it remains vulnerable to Layer 7 DoS attacks, especially handshake flooding. Recent work also proposes a fix: integrating crypto challenges directly into the retry token mechanism. This forces attackers to solve computational puzzles before the server commits resources.
What Is Transport Layer Security (TLS)?
Evolved from the SSL protocol, TLS is a cryptographic protocol designed to secure communication over a network and address various vulnerabilities. In earlier versions of TLS (1.0 to 1.2), the handshake process involved multiple round trips between the client and server, making it susceptible to attacks such as SSL negotiation and renegotiation. The separation between the key exchange and encryption phases also exposed these versions to downgrade attacks. Although communication after the handshake was encrypted, the handshake itself could be exploited—particularly through repeated renegotiation—which could overwhelm server resources due to its high computational cost. Below is an overview of TLS protocol versions and their security enhancements:
- TLS 1.0: Improved from SSL 3.0 by addressing known flaws, it retained similar key exchange and encryption methods. It remained susceptible to downgrade attacks.
- TLS 1.1: It improved security against Cipher Block Chaining (CBC) attacks by introducing explicit initialization vectors (IVs) and enhanced padding mechanisms, helping reduce vulnerabilities related to message tampering.
- TLS 1.2: It introduced cryptographic flexibility with support for stronger ciphers and hashing algorithms. It also enabled Perfect Forward Secrecy (PFS) through ECDHE. However, renegotiation and downgrade vulnerabilities were still possible, depending on server and client implementation and configuration.
The primary critical vulnerabilities associated with TLS protocols are related to renegotiation vulnerabilities:
- CVE-2011-1473 (OpenSSL Renegotiation Vulnerability): This vulnerability affected earlier versions of OpenSSL (prior to versions 0.9.8l and 0.9.8m through 1.x), where insufficient restrictions on client-initiated renegotiation in SSL/TLS allowed attackers to repeatedly trigger renegotiations within a single connection, leading to a DoS attack. The success of such an attack largely depended on the server and client configurations, the specific software version, and whether client-initiated renegotiation was permitted.
- CVE-2011-5094 (Mozilla NSS Renegotiation Vulnerability): This vulnerability affects Mozilla Network Security Services (NSS) versions 3.x, where misconfigurations allow attackers to launch DoS attacks through client-initiated renegotiation within the same connection.
What Is a THC-SSL Attack?
This SSL renegotiation attack exploits the server’s support for client-initiated renegotiation over a single TCP connection. An attacker repeatedly renegotiates encryption keys—an operation that is computationally expensive for the server but requires minimal effort from the client. This gives the attacker a direct path to exhaust the victim’s CPU. The attack requi)es only a single PSH-ACK packet without initiating a new TCP/SSL connection, making it highly effective with minimal resources. Each key exchange consumes approximately 15 times more server-side computation than on the client side, quickly overwhelming the server’s CPU and saturating the TCP session table as each handshake initiates a new session on the SSL daemon. In short, SSL negotiation and renegotiation attacks exploit the compute-intensive nature of the TLS handshake process.
TLS 1.3 and Its Security Improvements
TLS 1.3 is the latest version of the protocol, offering enhanced security and performance through several key changes—most notably, the complete removal of renegotiation. A server receiving a CHLO message outside the initial handshake will terminate the connection. TLS 1.3 enforces PFS using ephemeral key exchanges, ensuring that past sessions remain secure even if the server’s private key is compromised. It also streamlines the handshake process by eliminating unnecessary steps, and the CHLO message is used only once, preventing tampering through renegotiation. By removing renegotiation entirely, TLS 1.3 closes a major attack vector that was present in its earlier versions.
Remaining DoS Vulnerability
Despite the improvements in TLS 1.3, servers remain vulnerable to DoS attacks through handshake flooding. By initiating a large number of concurrent full TLS 1.3 handshakes, an attacker can exhaust server resources such as CPU and memory. Each handshake requires the server to perform costly cryptographic operations and manage the session state, highlighting that even the latest protocol version has limitations under high-volume attack scenarios. Repeated CHLO messages force the server to negotiate encryption parameters, generate cryptographic keys, and allocate memory for session states, even if the sessions are never completed. Additionally, cipher suite negotiations and interspersed, incomplete application data transmissions add further strain, compounding the CPU and memory load and making the server susceptible to resource exhaustion.
Example 6: TLS DoS attacks performed using some Kali Linux tools did not succeed when renegotiation was disabled.
Example 7: Many established connections in the terminal screenshot proved that TLS DoS attacks could succeed using OpenSSL tools, even when renegotiation was disabled.
TLS DoS Attack in Practice
Anyone can launch a TLS DoS attack using OpenSSL and just a laptop, but multiple machines are typically required to cause significant disruption.
Example 8: A practical DoS test was performed against a target server, where TLS 1.3 was the supported version, as indicated by the server’s responses.
Example 9: One terminal window ran a TLS DoS attack using a custom OpenSSL-based script. The second showed the current number of established connections, while the third displayed a live server overload by comparing baseline RTT with current RTT; the attack was conducted against michalsoltysik.com. Two of these terminal windows were outputs from Python scripts run in the operating system (Sołtysik, 2025a).
Example 10: TLS DoS attack increased CPU usage, inflated RTT delay, and pushed the server toward saturation, with OpenSSL connections stressing michalsoltysik.com. Note that both the attacker and the server could influence RTT due to CPU usage, network congestion, NIC (network interface controller) queue saturation, and handshake complexity.
Example 11: A YouTube demonstration shows a Layer 7 TLS 1.3 DoS attack using OpenSSL and Python on Kali Linux.
The Case for a Fully Secure Mechanism
Each protocol mentioned in the article was shown to suffer from handshake abuse, packet fragmentation, and/or amplification risks during the webinar’s live demonstrations (Sołtysik, 2025c). Effectively addressing these issues requires a fundamentally new approach focused on eliminating root vulnerabilities rather than relying solely on external mitigation.
Key principles for future protocol design should include:
- Stateless Session Management: Avoid stateful handshakes that force servers to track every session. Instead, use stateless cryptographic validation, storing minimal information until a session is fully established, reducing exposure to session exhaustion.
- Efficient Handshake Design: Simplify handshake processes to lower server-side computational costs. Lightweight handshakes that minimize server-side computational overhead and streamline cryptographic operations can effectively mitigate handshake flooding without compromising security.
- Early Client Verification: Introduce early-stage client verification to validate requests before encryption, enabling detection and filtering of malicious traffic without engaging full server resources.
- Throttled Resource Allocation: Dynamically manage server resources based on real-time demand, limiting per-session usage during traffic spikes to prevent overload rather than reserving fixed resources.
Stop Layer 7 Attacks: Train with the Certified Ethical Hacker (CEH)
To stay ahead of these complex attacks in the application layer protocols, cybersecurity professionals must go beyond surface-level knowledge and dig into the inner workings of these protocols and their misuse. The Certified Ethical Hacker (CEH) program offers this depth-bridging knowledge with real-world practice to equip cybersecurity professionals with the skills to detect, analyze, and defend against such sophisticated threats before they take down critical systems.
CEH covers the core domains of cybersecurity with AI skills mapped to all phases of ethical hacking. Through its immersive labs, real-world threat simulations, and a 100% practical exam, CEH turns knowledge into job-ready skills.
With year-long CTF challenges and scenario-based ethical hacking assignments, you will be prepared to identify and stop attacks even those in the design of web protocols.
References
- Sołtysik, M. (2025a). Cybersecurity Content. https://michalsoltysik.com/cybersecurity-content/
- Sołtysik, M. (2025b). How Web Protocol Weaknesses Enable Layer 7 DoS Attacks. YouTube. https://www.youtube.com/watch?v=_xP7_BnZCts
- Sołtysik, M. (2025c). How Web Protocol Weaknesses Enable Layer 7 DoS Attacks. EC-Council. https://github.com/MichalSoltysikSOC/Cybersecurity-content-videos/blob/main/presentations/Micha%C5%82%20So%C5%82tysik%20-%20How%20Web%20Protocol%20Weaknesses%20Enable%20Layer%207%20DoS%20Attacks.pdf
About the Author
