Yesterday evening, after a long time, a DDoS attack appeared on our internal common communication and drew everyone’s attention. Some of them solved the problem, others kept an eye on the situation to know what to write to customers…
This kind of attack was not so new. We have been dealing with it on a smaller scale in recent days. It didn’t cause too much trouble, because it didn’t cause prolonged service outages, but only momentary partial unavailability, and still to a negligible extent.
This was a very specific ACK flood attack that is used to overload the target’s webserver. Generally, an ACK (or other similar types of SYN+ACK) attack works by the attacker sending spoofed packets to a very large number of servers, which then send a response to the target of the attack. The target then has to deal with a large number of packets from different IP addresses until it runs out of space in the various tables, which usually means unavailability of the service.
In our case it was a little different. The big attack started around 9pm, when some web hosts experienced a 22% packet loss. It was impact. A few seconds of trouble and then tens of seconds of peace. At first it seemed to be a problem with the proxy servers that couldn’t handle the load. Which can’t happen. Each proxy server at webhosting is set up so that in case of malfunction it restarts and its role is taken over by another. In addition, if the proxy servers can’t keep up, another one will be created and started. That’s the beauty of WEDOS Cloud🙂 Simply unlimited infrastructure scaling as needed.
After a few minutes, the technicians were looking for a hidden attack so they could adjust the filters. Which is not easy, because we or our customers are always under attack. We’ve had attacks that have lasted dozens of days.
When it was obvious that it was not going to be easy, we turned on the tightening mode. This includes, for example, blocking ICMP packets, which is PING, and blocking access from certain countries (China, Russia, South America, etc.) This temporary measure minimized the impact of the attack, but customers who have availability measurements via PING or from one of the blocked countries received an unavailability warning, even though their service was working normally and they were not affected.
Meanwhile, our security experts have analyzed the attack. For example, they found that 627 packets out of 1,000 were zero length, which clearly pointed to ACK’s small-scale attacks of the past few days. Very clever, though. To evade detection, the attacker sent packets randomly and across all servers. For a while a few hundred packets went to the destination and then suddenly maybe 15 thousand per second and the end.
This was followed by a modification of the rules for DDoS protection to make it more sensitive and an increase in the number of proxy servers to distribute the potential impact load among themselves. We simply added more proxy servers and the attack was easier to resist and we had peace of mind.
Beyond that, it’s probably not interesting. The attack was repeated around 11 pm and 1 am.
A little behind the scenes of our protection
We have several elements in our network infrastructure that we use to detect and eliminate DDoS attacks. Our goal is to protect not only our network infrastructure, but also your services.
In the first line are “coarse filters”, these are powerful servers (and equipped with many 10GE cards) that just filter selected (compromised) traffic. They have nothing else to do. For example, there are completely “ordinary” blacklists. Simply, once an IP address is blacklisted, all traffic from it is discarded. IP addresses get blacklisted in different ways. For example, they have a bad reputation or are added based on current attacks, or they are IP addresses that we get from various blacklists (including paid ones). The “coarse filter” is also responsible for filtering countries and continents in case of massive attacks. At the same time, there are additional protection features such as various filtering by synproxy.
Another part of the protection we use is aimed purely against DDoS attacks. The relatively complex system uses several probes, which are very powerful servers placed in the network at different locations. These probes are used for traffic monitoring and subsequent evaluation. The probes are both at the entrance to our network and inside our network, and each filter has its own probe (both at the entrance and at the exit). So we have a detailed overview of traffic across the network.
If the probes detect an attack according to the set parameters, they send this information to the central system, which carries out the defined measures automatically. Most often, suspicious traffic is redirected to the so-called. switch. The detection detects this within a second at most and the traffic is redirected almost immediately.
Infected data does not “flow” directly to the server that is the target of the attack (and to the rest of our network), but is routed through special filters. Filters are very powerful servers with high computing power. Their task is to examine the traffic more thoroughly and then filter it. Everything is fully automatic.
However, filters do much more to avoid overloading our infrastructure. For example, they can block suspicious network traffic by country of origin of the IP address, filter TCP-SYN attacks and many other things. Normally these filters work with up to several thousand rules.
Also when operating through this switch we can set ICMP protocol restrictions (uses ping). PING is very often used to measure server availability. However, some customers inappropriately use it to measure web accessibility. In fact, if is under heavy attack and traffic is flowing through a switch where ICMP (PING) is disabled, the monitoring will start reporting an outage. The website continues to function.
Not only for this reason, it is better to use HTTP(S) requests sent to the web and check the return code for monitoring. Many things can happen when PING will report availability but the site will not work (failed automatic update, infected site and its redirect, congestion, shutdown, etc.).
We have additional protection against web servers. Online filtering, which has various functions and then IDS/IPS protection.
Part of our comprehensive protection is therefore also IDS/IPS (detection and prevention system), which is placed in front of the web hosting servers. It also monitors the content and method of communication, which it then compares with databases of the most common threats. For example, if someone tries to exploit a hole in your editorial system and our IPS/IDS system knows about the hole, it will block them. Currently, this protection is available for all web hosts. It works only on http traffic. We are preparing a change to make it functional on https. Just for the sake of interest, we add that on average throws away over 40% of the traffic that goes to web servers. Nobody misses him. It is for example scanning or various robots and attacks.
We also use the so-called ratelimitter, which limits the number of connections to the server, to a specific web host and to the visitor’s IP address for a certain period of time. This way, we have no problem blocking the offending communication while maintaining accessibility for the remaining visitors to your site.
All our protection is quite complex, but more or less automatic. As a result, we run it on more than 20 servers. If necessary, we can add more at any time to increase the filtration capacity.