On the night of 18.02.2022 to 19.02.2022, we experienced the strongest DDoS attacks on our infrastructure since April 2021, when we were hit by probably the strongest attacks in the history of the Czech Internet. Back then, for a brief moment, it managed to clog three 100 Gbps routes. This time it wasn’t so much about power but length, and apart from a few customers, no one noticed the attacks.
The first sign of the attack was recorded on Friday 18. 2. 2022 after 23:46, when the number of packets on the three main routes increased from 340,000 per second to 2.7 million packets per second (pkt/s).
From the back analysis of the sensor data (a cluster of servers designed to detect and analyze attacks), we then found that it was a combined attack via TCP and ICMP packets, which at peak reached over 3 million packets per second. Unlike routes, measurement and evaluation is done in a different way and everything that goes through the network (including backup routes) is included. It is also more accurate because it is based on existing data.
After midnight, the attackers managed to push nearly 800 thousand requests to our proxy server for a few minutes (before that, apart from DDoS protection, there is also a SYN filter that blocks attacking IP addresses). Basically, it was a primitive attack with common application layer (layer 7) requests. If our proxy system detects that it is under load, it will immediately start creating more proxies and spreading the load. The advantage of using a cloud solution.
For a short period of time, some customers experienced a slowdown in service (if their website went through the proxy servers that handled the attack). Overall, however, it was a slowdown from hundreds of ms to about 2 seconds per request.
This attack lasted only a short while and basically resolved itself. Several proxies were created to help clear the traffic and the SYN filter detected suspicious activity from new attacking IP addresses and restricted them.
Of course, my colleagues in support noticed that something was happening on the monitoring and pointed it out. The technician who was on duty checked everything. In addition, there were still some people online who have access from home. Including the developers. After a little debate, we thought it would be calm, but the strongest thing was yet to come.
It started with a “weaker” attack of 58.3 Gbps and 6.3 million packets per second. An attack like that would just clog the line for most of our competitors. The 100 Gbps lines we use are only used by specialized service providers who need to transfer large volumes of data or by operators of the largest data centers. Fortunately, we have 3 lines and backup routes.
The attackers tried different kinds of attacks with different strengths, but the real “taste” came after 1 a.m. The strongest attack reached 133.5 Gbps (it was several composite attacks). These are per-minute averages, so the offensive ceiling was stronger, but it doesn’t compute any better on routes.
We assume it was an attempt to clog the 100 Gbps line where the attacker hoped we had a cap. These attacks were already known and there was a brief slowdown in service before security dealt with it. In order within 6 minutes (see graph below in the evaluation).
What was interesting was that when the attackers couldn’t make it through brute force with one strong attack, they tried four longer attacks that exceeded 90.5 Gbps. These data are averages and therefore the peaks were much larger.
As far as packet count was concerned, the processors in the sensor and flirt servers definitely had a lot to count. The strongest attack had 13.5 million packets per second and the next 4 weaker ones had 8.7 million packets per second.
Then it was quiet.
Most of the attacks came from abroad. However, a relatively large part of it is also from the Czech NIX, which can be, for example, infected devices in the Czech Republic. A lot of companies rely on shutting down foreign countries in an emergency. Well, we have some bad news for you. A 20 Gbps attack can be done from the Czech Republic.
Source : https://www.nix.cz/ports/ports/day
Evaluation
After the very strong attacks last year, we upgraded our protections. We have been increasing the number of machines as well as adding memory and computing power to probes and filters. It was obvious now. Most traditional DDoS attacks were detected within 1-3 seconds and the problematic connectivity was immediately redirected to the filter. We almost survived the whole attack without any problems. We also found a few things to improve.
The following chart shows all the requirements for web hosting, WMS and WEDOS WebSite. After midnight there is a big increase in requests, which are proxy attacks (shown in light green). As you can see, it didn’t cause any slump.
However, a large attack with a measured strength of 133.5 Gbps (over 250 Gbps at peak) and 13.5M packets per second already does. On the minute graph you can see that during a total of 6 minutes there was a decrease in requests to the servers. The offense certainly didn’t knock us out. Moreover, these requests were not necessarily lost. Only slower, so they could not arrive until the next minute. Even so, there is a definite drop. But it’s a great result for an attack exceeding 100 Gbps, isn’t it? 🙂
It will work without losing requests, but we need to move it up a level
To be able to handle similar, or even much stronger, attacks without major impacts, we need to take our protection up a level. The decentralised WEDOS Global network that we are starting to build will be used for this purpose.
During the spring, we would like to deploy 25 HPE Moonshot 1500 server enclosures in 25 global datacenters. There are 45 physical servers and 2 switches in each HPE Moonshot 1500. In total, we will deploy 1,125 physical servers around the world. These servers will then handle and filter the traffic locally and everything will go to Hluboká filtered.
We currently have 2 of these boxes in operation and are preparing to send and deploy more.
We already have signed contracts, so in March we will start sending the first HPE Moonshots to the world. The value of hardware, multi-year contracts and other things around exceeds 100 million CZK. This is the largest investment since the construction of our second private datacentre WEDOS DC 2.
At the same time, we are finalizing the new WEDOS Global Protection service, which will be able to protect our customers’ websites. Currently, it already protects dozens of websites from L7 attacks. But it can do much more (various filtering/protection by country, extended protection for editorial systems, etc.).
Given the current situation, we are doing everything we can to accelerate the construction of WEDOS Global as soon as possible.
Finally, we will soon bring you an article describing how WEDOS Global protects a website under attack with over 70 million regular accesses per day. We’ll also write about how WEDOS Global defends a website that is under attack at over 13 million packets per second. Everything without the slightest hesitation and the site is up and running.
But next time…