As awareness of WEDOS Global spreads and more and more new customers who have a problem with DDoS attacks come to us, the more frustrated the attackers become. This leads to more intense attacks on our website and infrastructure. Sometimes they try really hard, earning us valuable data and experience 🙂 .
We can test new methods of protection against DDoS attacks on our own website, investigate their effectiveness and the load on the target server. 20. 11. In 2023, we tested a new form of protection that works great against “dumb” L7 DDoS Flood attacks.
What is an L7 DDoS attack?
HTTP L7 Flood attack is a form of DDoS that targets the 7. layer of the OSI (Open Systems Interconnection) model, i.e. the application layer. This type of attack is particularly insidious because it uses legitimate-looking HTTP requests to overload the target web server or application. Attackers generate a huge number of HTTP GET or POST requests that can be disguised to look like normal user traffic, making them difficult to detect and block.
Manifestations of an HTTP L7 Flood attack include significant slowdown or complete failure of the web service, abnormally high server resource usage (CPU, memory), and an increase in the number of HTTP requests logged in log files, often from several thousand IP addresses and hundreds of networks. These attacks can lead to partial or complete disruption of services.
Blocking HTTP L7 Flood attacks is a challenge for several reasons:
- Masking as legitimate traffic: attacks on the application layer often look like normal user traffic, making it difficult to distinguish them from legitimate requests.
- Application diversity and complexity: Layer 7 applications have different functions and characteristics, making it difficult to develop a unified defence strategy.
- Low attack predictability: attacks can be highly variable in volume, duration and techniques used, requiring adaptive defence mechanisms.
For us, detecting L7 DDoS Flood attacks is technically easier because the reverse proxy between the user and the target server can see the traffic. These proxy servers can analyse the traffic in detail, and even enter it, which is the basis for protecting against some more advanced forms of L7 attacks.
For example, we use this to treat new WordPress vulnerabilities, where the attack exploits an untreated input. Last time we manually added a rule to eliminate the risk of a new vulnerability in the WP Fastest Cache plugin for WordPress (CVE-2023-6063). An attacker could perform an SQLi attack on any application using this plugin using a modified cookie. We eliminated the request containing a cookie with SQLi.
So we have a very detailed overview of every request directed to the customer’s protected website, which goes far beyond the usual logging (access log, error log), and we also have a really wide range of options how to react. When you combine that with aggregating data from all web hostin services and protected sites, our ability to detect brand new threats, vulnerabilities, botnets, and so on increases manifold.
How (not) to block an attack
In October, we started working intensively on caching the content of protected websites. Then in November, we deployed caching for everyone, which led to websites speeding up dramatically. There’s a huge difference between caching content directly on the hosting or on our reverse proxies which are designed for that purpose.
Reverse proxies are always closer to visitors – we have servers in dozens of locations around the world and are gradually connecting them into local and national Internet Exchange Points (IXPs). So our reverse server is directly connected to your customers’ ISP. In addition, we have dedicated caching servers on site that use only software dedicated to this purpose. It makes a big difference if the cached content is returned by your webserver or by our dedicated reverse proxy.
Expressed in numbers, it’s tens to lower hundreds of ms vs 1 – 2 ms. Additionally, even cached content on your webserver costs computing power and connectivity, which you don’t have to worry about with us. We don’t care if we return ten or a million cached pages.
But back to the topic. After debugging the cache, we noticed that returning cached content is much cheaper in terms of server resources than, say, a captcha which we have to generate. The same is true for other types of protection that work with, for example, redirection. So what about just returning cached content to the attackers, instead of blocking them?
We conducted this experiment on Monday November 20. 2023, when our website and one subpage were attacked. The attack pattern of “dumb” L7 DDoS attacks has been the same in recent days. The main page of our site and some subpages, occasionally spiced up with parametric attacks. If you’re using WordPress, it’s easy to fight back.
The attack lasted about five and a half minutes. A total of 8.8 million requests came in from approximately 2,000 unique IP addresses. Peak 2.4 million per minute. These are just the requests that went through blacklists, various limiters and filters. Usually it is only a fraction of the original attack, usually 1/3 to 1/10. It depends on the type of botnet. We treat mobile operator networks differently from server provider networks.
What you see on the chart has arrived at the WAF. There are a number of filters and rules to deal with it. Let’s take a look at what was happening on the backend server. I mean, what went through to the web hosting – yes our main site is running on our NoLimit web hosting :).
On the top graph you can see the number of requests processed. In green are 200 (page return) and in blue are 3XX (301 redirect and 304 …). As you can see, our customers are browsing our website, as if nothing was happening. A few requests came through from the attack, but those aren’t even worth mentioning.
The bottom graph shows the length of the server response in ms.
As of approximately 21:43:40, an attack on the main page was in progress. The normal rules apply there. We filter the attack, and what gets through is usually not worth talking about, because the web can handle it (a few dozen requests per minute). In addition, the rules get stricter with each repeated request for individual suspicious IP addresses. Most requests are blocked on filters with limits. For testing purposes, captcha is disabled in all locations.
From about 21:47:15 the attacker changes their strategy and attacks a subpage of our website. There, we don’t put anything in its way, just the cached version of the site.
At this point, each site that receives a request for a subpage stores a static fingerprint of the subpage in its cache and returns it in 0.001 to 0.002 seconds. If the cache holds it for more than 3 seconds and receives the request again, it asks the server if the cached version is valid (request with status code 304). That’s why you see the increase in the 3XX requirements there. If we edit the page in the meantime, it will download the new version. Otherwise, the cache keeps serving the original content.
304 requests consume a negligible amount of computing power compared to returning a normal page (status code 200). This can be seen in the server response rate graph. Compared to 200 responses that take 50 – 70ms, 304 returns in 3 – 5ms.
Of course, we can change the parameters. We don’t need to check if you have a new version of the site every 3 seconds from each location. In the case of an all-site attack, this behavior can run into requests-per-minute limits for weaker hosting services. It is not a problem to set 60 seconds or more. There is always the option to clear the cache manually in the WEDOS Global administration if you need the current content immediately.
Two minutes later, the attacker gives up and ends the attack. Why waste his botnet’s precious resources on a target that doesn’t react to an attack with any protection, and won’t even slow down. There are plenty of targets elsewhere that can be shut down with the same amount of force and kept off-line for hours.
Conclusion
The new form of protection has premiered and we can continue to test and improve it. But it’ll be deployed soon. It’s worth it for everyone. This is easiest for WordPress sites and then of course customers with static sites.