On Monday 17.02.2020, thousands of our customers’ websites faced a massive security vulnerability that could have completely wiped out their websites in a split second.
Or one detective story from behind the scenes of WEDOS. No censorship.
Internal #distruptions communication channel for reporting security incidents and technical issues Monday 17.02.2020
19:46 Customer Support 1
He’s already writing 4. a customer within 30 minutes that his default template jumped on WP without doing anything
19:47 J.G.
(company management)
:
Where? What server?
19:48 Technician on duty in DC:
domain-A.cz hc1-wd50
(server designation)
, domena-B.cz hc1-wd53, domena-C.cz hc1-wd54, domena-D.sk hc1-wd63
19:49 Customer support:
domain-E.info wl39-f167
19:50 J.G. (for information we add that these are the initials Josef Grill):
I’m afraid someone hacked their template – they all have . Theme By ThemeGrill. But I had nothing to do with it 🙂 It would take some digging to find out what and how it was attacked or how to protect it
19:51 Customer support 1:
subdomain.domena-F.eu wl33-f208
19:55 J.G.:
Try to analyze the whole thing and find out as much as possible – even from the logs. It’s possible they had some kind of password…
20:04 J.G.:
Try something from the logs. Kibana, etc. When was the last time someone did an upload etc..
Gradually, other colleagues (customer support, technicians, developers) are joining. Those who have access to Kibana (an open source visualization for Elasticsearch), where data is collected in real time, including logs from hundreds of servers, are looking for attacks. Technicians with access to IPS/IDS look for patterns and intercepted attacks. Customer support collects information from customers and reviews the status of compromised sites.
20:09 Customer Support 2:
It reached into the databases and scrubbed them. Most people received an email about a new WP installation
20:15 Customer Support 2:
The FTP probably left everything, it just threw the databases to the default state and sent everyone an email from WP about the new installation
20:18 J.G.:
And what did it do? What script? What IP?
20:39 Customer Support 2:
But the user accounts are the same, nobody deleted them. Really only the posts and pages are gone.
20:40 J.G.:
It wants to find what IP or what script did it and block it everywhere. And for those people to make backups of the DB backups 🙂
Try to search now with the highest priority. I’m gonna have to track it down somehow.
There must be some IP somewhere in the log that logged into the administration or did some action via some URL
Do you know when that is exactly? Time. To the minute exactly and find it accordingly.
20:49 Customer Support 2:
hosting domena-F.eu db: d49XXX_xxxxxxxx Ran: 18:11:51
20:55 Customer Support 2:
Directly from the DB of the affected site, the time of creation of those default posts. coincides with the approximate time when people started writing
21:02 J.G.:
There is only this one in the log suspicious: 93.113.111.193 https://www.abuseipdb.com/check/93.113.111.193 but that’s later
21:05 Marketing 1:
And the database is empty (just the basic installation)? Couldn’t it maybe just create tables with a different prefix?
He has the files on disk http://….
21:06 Customer Support 2:
No, I checked. It just deleted the posts and the rest of the site was untouched + everyone got the email.
21:02 J.G.:
Do you still have time for another site?
21:13 Marketing 1:
Ok so it’s a worldwide problem: “The developers of the ThemeGrill Demo Importer for WordPress have updated the plugin to remove a critical bug that gives admin privileges to unauthenticated users.” “Once all tables have been dropped, it will populate the database with the default settings and data after which it will set the password of the “admin” user to its previously known password.”
21:14 J.G.:
So I hit it right away 🙂 But I had nothing to do with it.
21:14 J.G.:
I would probably send it through downtime to all web hosting clients.
21:14 J.G.:
Can you put together a simple text to send to everyone?
Plus I would look for more times to find more information and try to find some IPs etc.
And we’ll put it on social media
But with a note that it’s not my fault 🙂
21:18 Marketing 1:
I’m on it.
21:25 J.G.:
Do you have any other times? I can’t find anything – how they got there.
At 21:43, a warning was posted on Facebook and Twitter describing the problem and linking to the security report of the team that discovered the vulnerability. At the same time, an email was sent to all customers. At that time we did not know how big a problem it was. Initial estimates, however, pointed to a few percent of customers being affected.
21:43 J.G.:
And I still have a feeling that the users will do it themselves. When they do something in the administration, it gets executed or activated.
21:57 Customer Support 2:
Someone wrote that he hadn’t been in WP administration for a week and one lady wrote for a month
22:09 J.G.:
We analyzed the security hole and found the first attacker 45.129.96.17
At this point, we already knew exactly what the attacker was sending to vulnerable WordPress sites to achieve a “reset”, so there was no problem identifying him. At this stage, we performed the first hits manually and blocked the first IP addresses.
22:21: J.G.:
According to kibana, there are about 150-200 hacked sites
The first automated IPS/IDS protection scripts were launched that could detect attackers and automatically block their attempts on HTTP sites. At the same time, the database for the second level of protection, which also protects sites on HTTPS, started to fill up. It parses logos and searches for new specific strings. Within a minute, everything is protected.
23:07 J.G.:
There are other attacks. Now for example via IP 2607:5300:61:bd9::107
So what happened
By the time we figured out what was going on and set up IPS/IDS protection, 154 sites were affected. The attacker “reset” WordPress through unpatched access in a leaky plugin. He had it pretty easy. Details of how this was done can be found in the article by the security team that discovered the bug.
From the perspective of our IPS/IDS protection, it was difficult to automatically detect the attack because the attack was performed via a WordPress administration URL call with the do_reset_wordpress variable.
After we found out what the security flaw was, we adjusted the IPS/IDS settings and saved more than 1000 more WordPress installations that have the problematic plugins installed by this morning.
At the moment (Tuesday 18:00) there are no more attacks. As far as we know, the attacker is already blocked. We block any attack that we detect over HTTP. Unfortunately, if it’s a new IP address and the attack is directed at HTTPS, we can’t help there for now. We can’t see encrypted traffic. But we will see soon 🙂 .
In the chart below you can see the statistics of attacks over time on the 10 servers that were the most frequent targets. We have hundreds of servers. The total number of attacks recorded was almost 41,000.
Just under 200 (about 183) were successful.
The attacks were carried out from the following IP addresses:
103.221.222.179
103.4.217.81
107.180.225.158
142.44.151.107
149.202.75.164
159.65.65.204
168.63.19.216
172.68.10.50
172.68.245.131
185.45.72.159
188.166.16.17
188.166.176.184
198.12.156.154
2001:41d0:d:34a4::
209.251.53.192
2607:5300:61:bd9::107
2a03:b0c0:2:d0::11f0:6001
45.129.96.17
46.101.174.128
50.63.162.9
51.68.124.88
62.76.187.179
68.183.204.202
The authors of the vulnerability information found IP addresses less:
45.129.96.17 200
185.45.72.159 4
46.101.174.128 2
50.63.162.9 2
51.68.124.88 2
62.76.187.179 2
68.183.204.202 2
103.221.222.179 2
107.180.225.158 2
142.44.151.107 2
We have a larger sample of sites because we host about 150,000 sites and a large portion of them (tens of thousands) use WordPress. And more than a thousand of them are using the contestable plugin. In total, 200,000 websites with this plugin were reportedly compromised worldwide. That’s nice, because almost 0.7% of all of them are with us. Quite a good share of the global number 🙂
What we are planning for the future to improve the safety of our customers
Security holes in content management systems, plugins and templates are not our fault. But we know they are there and will be there, which is why we are constantly improving our protections. This year we have two big innovations that will help in these cases.
The first is the enhancement of proxy servers that will allow us to filter and protect the content of encrypted connections. This will give us the power to detect and filter attacks that go even to HTTPS sites.
Another is the WEDOS AnyCast service. It will bring us new possibilities of protection not only against DDoS attacks. In this case, for example, we could add a captcha to the suspicious traffic, which would reliably stop the bots that were carrying out these attacks.
Conclusion
We received a large number of responses. Some accuse us of sending an “alarmist message”. At the time of writing, we did not know the exact extent of the attacks or the number of potentially vulnerable sites. We host several tens of thousands of WordPress sites. We made a quick decision and judging by the many reactions, it paid off. It was impossible to delay.
Before midnight we wrote to all clients whose websites were hacked to tell them that this had happened and to sort it out and how to proceed.
It took us several hours to find other vulnerable sites. Try searching hundreds of physical servers for certain files with certain attributes and certain content. You really can’t go any faster than that. 🙂
When we finished this morning, we had already informed all the specific customers concerned. We have written to everyone who had the plugin in a vulnerable version to address the situation.
We don’t know how anyone dealt with it, but most of them did.
Most of our websites run on WordPress. It is now the most popular content management system. So we went with that majority and sent an “alert message” to someone rather than delay and have an unknown (at the time) number of sites deleted. Take into account that we have about 150,000 tier II domains running on webhosting there, plus a relatively high number of subdomains on tier III domains…
It’s on hundreds of physical servers… And suddenly it is not so easy to find out what is running on WordPress and what has what plugin and in what version. Detection also takes quite a long time. The next day it took us a few more hours. If we had a similar delay, there’s nothing to save.
Yes, we do get negative or confused reactions from people who don’t use WordPress, but we answer them honestly. After reading the above lines, you will understand that it was about every moment. We couldn’t put anything off.
We have also received several inquiries as to why we were the only ones to bring the problem to the attention of our competitors, and why our competitors have not responded in any way. Actually, we prefer not to have an answer to that question 🙂
We have all data – logs centralized and so we log tens of thousands (or more) of records every second. This BigData helps to solve anything. You know all the context right away. We have detection and filtration systems, and we’ve dealt with it. We have a team of experienced colleagues who are able to deal with similar situations. That’s why we are where we are. 🙂
We knew that our customers’ websites were at risk, so we took action. We’re not ashamed of it. If someone “unnecessarily” logged into the administration of their editorial system to check if everything was updated, we apologize.
Honestly, it would be nice if everyone made all the updates so we don’t have to deal with a similar detective story next time.