When 162,000 unique IPs attack 143,000 websites

[gtranslate]

On Monday 23. October we experienced the most extensive attack on WordPress installations. The “Indonesian botnet” has fully awakened and started massively attacking the files of xmlrpc.php. Although the attack tried to look inconspicuous, the result was truly massive.

Indonesian botnet

Every day we do several data aggregations to identify new potential threats. One of them is searching for SQLi attacks. New botnets usually try to behave inconspicuously until they grow up. Those who control them thus use them to subtly find vulnerabilities for further growth. During October, we saw a fairly decent increase in activity from Asia, with the provider “PT Telekomunikasi Indonesia” leading the way. Hence the internal designation “Indonesian botnet”.

When we see a local increase in various malicious activities like this, it is usually related to the spread of some malware on mobile phones. This is often the case in areas where popular mobile apps are restricted for political, religious or purely commercial reasons and people download them from third-party servers where they may contain malicious code.

It then carries out attacks over the networks of mobile operators and fixed connections offered by Wi-Fi. When we aggregated the data for 24 hours, “PT Telekomunikasi Indonesia” was always seen at the top.

WordPress and xmlrpc.php

The xmlrpc.php file in WordPress contains a script to implement the XML-RPC protocol that allows systems to communicate with each other. XML-RPC is a protocol that allows simple remote procedure calls. In the context of WordPress, xmlrpc.php allowed external applications (such as mobile apps or other web services) to interact with WordPress, for example to publish posts, manage comments, and so on.

However, xmlrpc.php is commonly the target of attacks for several reasons:

Brute Force Attacks: attackers can use xmlrpc.php to perform brute force attacks on usernames and passwords, a method where the attacker repeatedly tries different combinations of usernames and passwords until they log in.

DDoS attacks: the file can also be exploited to perform distributed denial-of-service (DDoS) attacks, where an attacker calls xmlrpc.php with a large number of requests, overloading the server.

Attack amplification: xmlrpc.php can be exploited to amplify attacks, where an attacker sends a small request that leads to a larger response from the server, increasing the strength of the attack.

If you have your domain protected by WEDOS Global Protection, the xmlrpc.php file is protected by a series of comprehensive rules that do not restrict WordPress and at the same time fight off attempted attacks.

Monday 23. October 2023

After midnight, our automated external monitoring aggregating data by servers from all WEDOS OnLine sites detected minor slowdowns on some servers. The higher load was also confirmed by internal server load monitoring.

The night patrol made a few interventions that stabilized the situation. There was no crisis situation because we keep the servers semi-empty to allow enough space to cope with the surge load. In addition, there was night traffic.

The attacks continued, but the servers were not affected until the morning when the normal daily traffic started and reports from WEDOS OnLine started coming in that some servers were slower than normal. We started to look into it more intensively and analyze the traffic.

We were soon able to determine that this was an attack on xmlrpc.php. The IP addresses of the “Indonesian botnet” were known to us. So we did a bulk data aggregation, and even though the attackers tried really nicely to cloak themselves, they only used 6 common useragent.

Useragent is a short text string that a web browser or other client sends to the web server with each request, identifying its web browser, operating system, version, and other information. It is used to let the server know what type of device or application it is communicating with.

So it was enough to filter out the other useragent aggregations and leave only these:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36 Edg/89.0.774.77
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36

That’s when we first saw the extent of the attack. Despite the huge amount of requests, only units to lower tens of reuquests per hour were coming from one IP address. And they managed to send over 700 thousand requests per hour to some 143 thousand (sub)domains. If we didn’t download access logs from all of our servers to one central location and evaluate them in real time, we wouldn’t have a chance of tracking anything like this down.

As an administrator of one website, you have no chance to recognize such a distributed attack and, in fact, you can’t even defend against it, because each time a request comes from a different IP.

Since the customer sites were not crashing or slowing down in any significant way, there was no need to act hastily. So we divided up the work.

• Find out which sites suffer the most and move them to WEDOS Global Protection. If the customer has our DNS with the web host, we will do it automatically for them. These large-scale but primitive attacks will not harm sites protected by WEDOS Global Protection.
• All problem IP addresses that are the most active, but otherwise do not access our customers’ websites, temporarily put on the blacklist.
• Define clear rules on how to protect websites on web hosting with minimal impact on functionality.

At about 10:40 everything started to move. It was interesting to see how the “Indonesian botnet” reacted to this. Once we started banning his IP addresses, others started sending more requests. So the total didn’t drop much. However, after some 50,000 IP addresses, which we were sure no one would miss, it was clear that the attack was waning.

Then the engineer who is in charge of the web hosts and the proxies that help them with the load came up with a solution that implemented one of the rules from WEDOS Global Protection for everyone just on the proxy. It was a bit of a risk, so we preferred to put a warning on the status page. But nothing broke, and the attackers gradually started to return 403 (blue) instead of 200 (purple). We did not intervene where it was not possible, because the impact of such deviations on the servers was minimal. Customers with WordPress installations containing something that xmlrpc.php was slowing down significantly were put on WEDOS Global.

After the attack ended

Our new data analytics department was then tasked with reverse engineering the attack and identifying other vulnerable WordPress sites. These sites then received support to complete the move to WEDOS Global Protection.

And what are the final statistics?

Web hosts received: 17 771 841 requests
Total targets (sub)domains: 143 727
Total attacking UIP: 162 952
Total time to generate webserver responses: 3870 h 22 min.

The statistics do not include blocked requests to WEDOS Global Protection, blacklist and proxy.

As you can see, it was nothing terrible. That’s why we could afford such a lax approach. Fortunately, our servers have enough power and spare capacity to handle the excessive traffic. If things got tough, we would act faster and harder.

What about the botnet? He is still active. It is exploited for standard L7 DDoS attacks. He has carried out about a dozen of them by the end of October that are worth mentioning. It also still performs vulnerability scanning on a smaller scale, but uses a different useragent.

Protect your WordPress with the WEDOS Global Protection plugin