WEDOS Global Protection will be the first service built on WEDOS Global. We are currently in an open testing state. As part of the trial, you can add your domain to wedos.global for free and try different forms of protection and traffic filtering. By mid-October, the number of testers had surpassed 100. You do not need to have a domain or hosting with us. We just need to use our nameservers.
Now over 1,000 large websites and even one e-shop with a turnover of around one billion crowns use WEDOS Global. Every second we handle several thousand pages there.
Currently, the process of adding new domains is not yet ideal, as it is designed more for more experienced users – they can set it up to avoid downtime. We need to tweak it a bit to make it manageable for everyone and without failure (first we need to import/insert NS records and then if they are correct, set the DNS of the domain to ours). That was the way it was at the time of writing, but we are evolving and that is no longer the case. Now it is so that you enter the domain and we will arrange everything. You end up just changing DNS servers and everything goes from us, without any downtime. We also want to ensure the change of DNS servers eventually. But that’s a week away. For domains that have our DNS the process is not yet automated for security reasons and we will come up with a new solution next week.
Once you set up our nameservers for a domain, the traffic goes through WEDOS Global, which means you use all active points, of which there are currently 15. But we’ve bought the hardware for 50 points.
Current points:
Europe
- 🇳🇱 Amsterdam
- 🇪🇸 Barcelona
- 🇨🇭 Zurich
- 🇫🇮 Helsinki
- 🇨🇿 Hluboká nad Vltavou DC1
- 🇨🇿 Hluboká nad Vltavou DC2 (in oil)
- 🇬🇧 London
- 🇪🇸 Madrid
- 🇫🇷 Paris
- 🇸🇪 Stockholm
- 🇦🇹 Vienna
- 🇵🇱 Warsaw
Asia
- 🇭🇰 Hong Kong
- 🇸🇬 Singapore
- 🇯🇵 Tokyo
Today (4.11.) the hardware has been delivered to Sydney, so we will be up and running there in a few days. That will be 16. Location.
We have signed 5 sites in the USA, one in South Africa and one in Turkey. We want to launch it in November. We are also discussing Bulgaria, Oman, Dubai, Mexico, Argentina, Brazil, Colombia, Chile. If it works out, we’ll make it to the end of the year.
As far as protections are concerned, our existing DDoS protection against brute force attacks is in place at every point. There is also protection against traditional L3/L4 DDoS attacks. Then there’s our forward SYN filter, which blocks hundreds of thousands of problematic IP addresses. They can be there for a few minutes, hours, days or permanently. They are gradually added/removed by robots or colleagues based on traffic analysis from more than 150,000 websites.
This is a good baseline that will save you the cost of operation (customers who have protected VPSs have confirmed to us that they would be fine with a lower variant because we filter out all the overloaders and bots). This may come in handy given the rising cost of electricity (and consequently hosting services). We reckon that WEDOS Global Protection will cost “a few hundred” per month for businesses. If it saves them a grand a month, that’s all the more reason to use WEDOS Global 🙂
Other protection settings are up to you. You can set which countries, continents or IPs you want to block, or you can set a captcha or cookie test and redirect.
So basically it is a manual protection that you have to turn on. When an attack is launched at you, you have an immediate opportunity to act. For example, you set a captcha for everyone outside the Czech Republic/SK. In a few minutes, you’ll have peace of mind.
But we would like to take it further and create some kind of automatic protection model. If there’s an attack, the robot will take care of everything. With that said, we expect to deploy for live testing in a few days.
This is done by using different models to see how a filter would perform in real operation.
Benefits we didn’t have before
WEDOS Global brings two new advantages to deal with problematic traffic.
You have individual points that pull local traffic together and send it on. He is always something specific and you can treat him differently. By default, traffic from all over the world comes to your server and you have to work with what you know about it. That’s not ideal. According to the IP, you may know that it belongs to a Czech company, but in reality it is a foreign operation. At WEDOS Global, you can see this because the traffic takes the shortest route.
Take a look at the following table. All IP addresses “claimed” to be Czech. The Czech ones go only through Hluboká nad Vltavou and Prague.
IP addresses that go through Tokyo, for example, because that’s the shortest route for them, are suspicious and we may treat them differently.
The second advantage is that you don’t have to block suspicious traffic right away. It is enough to put an obstacle in its way, which can be easily overcome by a human, but not by an attacking robot. For example cookie + javascript redirection has worked well for us. This is not a problem for a web browser. It saves and verifies the cookie and learns from the javascript where to go. This user is flagged and does not have to repeat the test for several minutes (it is up to you how much time you give them).
Of course, there are advanced robots that can solve this, and even methods to get around it. Well, we can put a captcha on those. However, they are not used for massive attacks because it is more expensive than simply firing tens of thousands of requests per second.
How we will verify traffic
Measuring the anomaly across the entire traffic is not that easy, especially if you are used to high traffic. However, if you watch the traffic at each point separately, you can see the anomalies quite well. Most of the attacks come from all over the world.
This means that it is sufficient to measure each point separately. If your site is targeting Bohemia, we will see only a slight increase on the Hluboká and Prague points, but significant on the other points. On the other points we can let the cookie redirect. It’s nothing that would bother the average visitor and it stops the attacks. Slight increase from the Czech Republic the website will withstand. And if it wasn’t tiny, the protection would kick in.
In the following table you can see a simple table based on real traffic at one of the 15 points. In the first column are the domains, in the second column the traffic for the last 5 minutes, in the third column the traffic for the previous 5 minutes and in the fourth column the 5 minute averages for the last hour.
Let’s say an attack came in and the number of requests was maybe 5x the 5 minute average (with some minimum). So at that point we can just drop the cookie check + redirect.
This is not a simple concept at all. When evaluating, we can work with any data from the access log, including how long requests take. We can also do protection that will restrict traffic at that point because it takes a long time to process (attacking uncached sites). Either for the entire domain or for an IP address.
The possibilities are countless. We have lots of ideas and especially enough traffic to simulate it realistically.
In the future, the plan is to take this further using AI (machine learning).
What went wrong #1
Some attack detection methods were so good that they provided a decent list of compromised IP addresses. If you block such IPs, nobody minds and on the contrary you save a lot of server resources, because the IP belongs to a computer (server) in the botnet that is always doing something. He searches for vulnerabilities in the morning, tries brute force attacks in the afternoon, and spams comments at night.
One such “honeypot” tracked brute force attacks on the WordPress login page. It works perfectly. Every day it was dozens of new IP addresses involved in some botnet. Well, one day my colleagues from support wrote why we have blacklisted Uptimerobot, which is a service for monitoring the availability of services. That the repairman was conducting brute force attacks?
In truth, he fell there deservedly, but it was the user’s fault. We have found that several of our customers (really several) have decided to check the availability of the login form. We taught them all a lesson, but if you look at the stats, others have started doing it.
But this was more our fault. Firstly, the method of the request must be controlled. Thus, whether any data is sent to the form (via POST) and especially the IP addresses of services like Uptimerobot we must have on whitelists. Which, at the time of writing, we already have.
Due to the number of attacks on wp-login.php, all these files will be automatically protected on WEDOS Global Protection.
What went wrong #2
Over the holidays, we detected extensive botnet activity looking for vulnerabilities via SQLi. We even dedicated a blog article to this topic. The attackers used two methods directly from the compromised VPS and through popular VPN services. On some days it was even higher hundreds of thousands of requests from tens of thousands of IP addresses.
Since we were monitoring this really closely, in addition to gradually moving activity between different VPN and VPS providers, we tried a protection model on it that would make a filter for WEDOS Global Protection.
Well, Googlebot fell into it, or rather one of its scopes that was not on active whitelists. We were alerted by a customer, so he was only there for a few hours. Nothing that would affect the position of the site in search engines. Google expects that sometimes the server will reject requests.
These were the following requirements. At first glance, they really look like an attempt at SQLi.
And it came from these IP addresses.
This is a similar pattern of attacks that we have detected from botnets through VPN providers. Systematic testing on a relatively small scale, from multiple IPs on a single /24 range where browsers change. For the sake of completeness, browsers in the case of “vacation attacks” via VPN are automatically generated, so there would be a different one for each access.
These IP addresses belong to Google and are used by Googlebot.
And this particular one has a nice entry in AbuseIPDB.
This is indeed the official IP address of Googlebot, which you can verify directly with them.
So does Googlebot perform SQLi attacks? Most likely not. The attacker just prepared a URL for Googlebot to visit. Googlebot is whitelisted everywhere, so it can get away with this attack. And that’s what the striker is betting on. Pretty original, huh? 🙂
Next time we will write about what we have done well. We’ll show some examples of how WEDOS Global has helped speed up the web, improve accessibility, improve search engine rankings, and protect against attacks or save money on electricity or expensive hardware.
Conclusion
At WEDOS Global Protection we work on all fronts. We are looking for new locations, arranging with datacenters and connectivity providers in selected locations, configuring the HPE Moonshot 1500 to be sent to the locations, arranging for transportation and paperwork, on-site installation, developing the backend and frontend, fine-tuning the business model and of course designing protections. It is the biggest project we have ever worked on at WEDOS. And of course we are looking for other testers. So don’t hesitate to try wedos.global. You don’t need to have hosting or a domain with us, just use our DNS, which will also mask where your real hosting is.
Soon we plan to add more services based on our global WEDOS Global network. You will soon see DNS servers. This will make your domains and websites more secure and globally accessible, and will provide faster and more reliable responses. We’re already testing this in live deployment.
Then we want to add intelligent mail filtering and spam protection. Development is already underway there.
Later this year, we will deploy WEDOS OnLine – our monitoring system that will have probes all over the world.
What’s next? We are planning private VPNs and much more. Let yourself be surprised. We are a Czech company with two datacentres in Hluboká, but we want to be a global company with global coverage and global offer.