We’ve written a lot about our protections. We wouldn’t be here without them. We are constantly improving them and trying to stay one step ahead. However, the progress we made last year was really significant and, as it turned out at the end of the year, very important.
Current status of protections
Currently, websites and our infrastructure are protected with 3 levels of protection.
- DDoS protection – protects mainly against brute force attacks
- IPS/IDS protection – smart protection that filters malicious network traffic
- SYN Filter – robust protection that blocks or limits problematic traffic from hundreds of thousands of IP addresses based on rules from our logs and other sources.
On top of this, we have an additional layer of protection that is still under development that protects customers from specific threats. It will be part of our WEDOS AnyCast solution. The advantage is the very fast deployment and the possibility to introduce security features such as captcha from our side.
You can learn more about our protections in our boss’s talk at OpenAlt 2020.
How to subtly crash the web
In November, we noticed a new interesting attack. One organization was under a hard to track attack, they had servers elsewhere, and they asked us for help. We offered to deploy our new protection and it helped.
We couldn’t analyse it in detail at the time because we didn’t have the logs (we didn’t have access to their server). So we deployed our new solution and suddenly we saved their “hide”. Their e-shop was up and running. Moreover, it was a rescue in a very serious situation (the whole organization, used to an offline environment, suddenly depended only on online income) and in the pre-Christmas period.
However, this changed in January when one of our customers on NoLimit web hosting became the target of a similar attack. His site is still on older servers that don’t have proxies and use slower processors. However, its normal traffic of around 60 – 100 thousand requests per hour can be easily supported. But then suddenly there were many times more demands. It was over a million requests in 10 minutes and each was from a different IP address and directed to different URLs.
These were not spoofed IPs, but actual requests from the IP addresses maintained. There was real (and two-way) communication within the requirements.
These requests were very evenly distributed. During the 4-hour attack, no more than 600 requests were made from a single IP address to a single domain. These were mostly in the lower hundreds. At the same time, the requests were directed to different sites. Of all the attacking IP addresses during the entire attack, one did not visit a single site more than 4 times. The average was 2 visits to the same page in 4 hours from 1 IP address.
Accesses from 1 IP looked something like this:
Individual IP addresses had different browsers, operating systems and came from computers and mobile phones.
From this behavior, we assume that the compromised devices were part of a botnet.
The most active IP addresses that participated in the attack:
| IP address | Number of | ISP |
| 222.135.231.178 | 1297 | JINAN Xinhaikeji Net Bar |
| 119.116.186.28 | 1234 | China Unicom Liaoning Province Network |
| 119.116.183.9 | 1202 | China Unicom Liaoning Province Network |
| 112.48.9.38 | 1099 | China Mobile Communications Corporation |
| 61.240.226.52 | 1090 | China Unicom |
| 112.48.9.18 | 1049 | China Mobile Communications Corporation |
| 112.48.9.53 | 1003 | China Mobile Communications Corporation |
| 119.116.191.98 | 928 | China Unicom Liaoning Province Network |
| 119.116.181.39 | 926 | China Unicom Liaoning Province Network |
| 119.116.189.18 | 876 | China Unicom Liaoning Province Network |
| 119.116.181.159 | 869 | China Unicom Liaoning Province Network |
| 119.116.184.89 | 840 | China Unicom Liaoning Province Network |
| 119.116.179.175 | 837 | China Unicom Liaoning Province Network |
| 112.48.9.91 | 797 | China Mobile Communications Corporation |
| 112.48.9.6 | 796 | China Mobile Communications Corporation |
| 222.135.230.133 | 785 | JINAN Xinhaikeji Net Bar |
| 119.116.181.132 | 775 | China Unicom Liaoning Province Network |
| 112.48.9.86 | 766 | China Mobile Communications Corporation |
| 222.135.230.122 | 748 | JINAN Xinhaikeji Net Bar |
| 119.116.179.110 | 734 | China Unicom Liaoning Province Network |
The customer used multiple domains (as aliases). The attacker was careful not to exceed 600 accesses from 1 IP address in 4 hours for any domain.
As you can see all IP addresses came from Chinese networks – mostly mobile providers. When we group IP addresses by /16, the attack becomes more visible.
Here is a list of the most active ones.
| IP address | Number of | ISP |
| 119.116.0.0/16 | 78884 | China Unicom Liaoning Province Network |
| 112.48.0.0/16 | 65868 | China Mobile Communications Corporation |
| 125.115.0.0/16 | 61714 | CHINANET-ZJ Ningbo node network |
| 183.27.0.0/16 | 56818 | CHINANET Guangdong province network |
| 220.175.0.0/16 | 48405 | CHINANET jiangxi province network |
| 141.101.0.0/16 | 44912 | WildPark Co (Ukraine) |
| 218.68.0.0/16 | 40428 | Tianjin Huaqing Trade Co., Ltd. |
| 58.214.0.0/16 | 35946 | Wuxi Jiangying Telecom Finance Dept |
| 101.17.0.0/16 | 34244 | China Unicom Hebei province network |
| 220.202.0.0/16 | 29725 | China Unicom |
| 125.123.0.0/16 | 27786 | CHINANET-ZJ Jiaxing node network |
| 183.250.0.0/16 | 25516 | China Mobile Communications Corporation |
| 150.255.0.0/16 | 24530 | China Unicom Hainan province network |
| 113.121.0.0/16 | 23735 | CHINANET SHANDONG PROVINCE NETWORK |
| 118.79.0.0/16 | 22350 | sxxz-erfenju-BAS (CHINA UNICOM China169 Backbone) |
| 39.184.0.0/16 | 21851 | China Mobile Communications Corporation |
| 39.181.0.0/16 | 21697 | China Mobile Communications Corporation |
| 221.197.0.0/16 | 19441 | China Unicom Tianjin Province Network |
| 101.24.0.0/16 | 17570 | China Unicom Hebei province network |
| 211.97.0.0/16 | 15672 | China United Network Communications Corporation Limited |
These are purely IP addresses that have already arrived at the server and the server has logged them. Some ranges have been “locked in” under other rules. However, as the attacker was careful, he avoided many filters. It should be noted that IP addresses in these ranges are in most cases clean and have not had a single report of an attack on the various blacklists in the last year, which is quite rare.
Again, we can speculate that this is some new botnet.
And what did it do to our customer’s website?
The following graph shows the average server response time to a request. You see a total of 3 attacks on it. The first was directed at two of his domains. When the attacker found out that we had deployed protection, he found another domain (alias) and launched a second attack on it. All within one NoLimit hosting. We’ve added that to the protection as well. The third attack (slightly different) came at night, but did no more damage. The customer had several dozens of domains in different TLDs and the attack was directed to all of them.
The following chart shows what this has done to his website. The purple color shows 503 errors that started to occur due to PHP thread exhaustion. It’s just logs from the server. At that time, hundreds of thousands of other requests were already being filtered on the protections.
This chart filters purely traffic from China.
How we protected our customer’s websites
Basically the longest it took to communicate and arrange everything. The customer used our DNS for most of the domains, so deploying protection from our side was not a problem. The only minor hitch is generating the Let’s Encrypt certificate. There’s nothing else to wait for. We can deploy this protection right away. The deployment took units of minutes and within about 30 minutes all DNS records were changed.
We texted and emailed the customer to let them know what was going on. We were in contact with him afterwards.
We immediately deployed captcha for traffic from China at the customer. From the statistics we later learned that only around 300 visitors completed the captcha in the whole day since the deployment. They got to the site normally. The rest were blocked on protection.
When the attacker found out that the main domains were protected, he tried even harder on other domains. So we gradually added all of them and it was quiet.
In the future, everything will be protected with a single click. It won’t even need to have hosting or a server with us. Just the domain and DNS will be enough. The price? We don’t know yet. The basic option could be 500 CZK/month.
When the service will be available
This protection should be part of WEDOS AnyCast, a project that is basically ready and tested and waiting for the team to finish and integrate it into our system.
Of course, the protection is already running in test mode. If you are under some interesting attack, you can write to us and we will be happy to test it on you 🙂
Conclusion
We have been seeing this kind of clever attacks more and more often lately. A few years ago, the attack was by force (Gbps or packet count). There was a competition to see who could send tens or hundreds of Gbps… That’s what our DDoS protection works for.
However, the trend has gradually shifted to clever and insidious attacks that are difficult to detect. We’re heavily protected by IDS/IPS protection. However, this is a new trend and current state of affairs.
But similar attacks are not just coming from China. This is more likely to be specific to the botnet in question, which acquires new zombie computers/mobiles in a targeted manner (e.g. fake applications targeted at a specific market). Last year, for example, we saw this kind of attack from Russia.
It’s only a matter of time (more like money and attacker contacts) before we see them globally. We want to be prepared for this and offer our customers the protection they need.