We have been under heavy DDoS attacks for the last 14 months, but we have made many adjustments to the network and built very robust AntiDDoS protection.
Note for introduction – The article below was broken down in late May 2015 but has not been published. It was finally finished in August. Just to give you an idea and an explanation. In May it was 7 months since we deployed DDoS protection and in 7 months we have had over 50,000 attacks. In August, it’s been 10 months and we’ve already surpassed 100,000 attacks. This clearly shows the growing trend and necessity to deploy DDoS protection.
Introduction to DDoS attacks on WEDOS
In October 2014, we wrote an article about what was happening with DDoS attacks in our country. Remember here. The original article is certainly worth remembering and worth reading.
The last 14 months have been challenging. Extremely demanding. We worked hard and it showed. The result is robust DDoS protection that we continue to expand and improve.
What we have experienced…
In 10 months and 1 week (since we have DDoS protection deployed and we are measuring the number of attacks) we have recorded over 100,000 different DDoS attacks (at the time of writing the article it was 100,473 attacks). Huge number. This works out to an average of 321 incidents per day. These are numbers that even we can’t believe.
In 7 months it was 50,000 and now in another 3 months it is also (another) 50,000. The number of attacks is growing, and their intensity is also increasing. At the same time, it must be admitted that we are also better at detecting attacks.
The strongest attacks were around 30 Gbps and almost 4 Mpps*. At the time, these were attacks that were among the largest (at the time) in Europe. Attacks in the order of Gbps units are quite common (and let’s say commonplace). Just at the time of writing, someone was plaguing the www.wedos.cz website with attacks over 3 Gbps. None of the clients recognized anything.
* The Mpps (Million Packets Per Second) value indicates the number of packets per second in millions.
What we’ve done…
Tests, tests and tests
We tested several solutions. We tested hardware, we tested services, we tested different software solutions and in the end we chose the (basically Linux) solution that has the most variability and performance and basically freedom of configuration.
Negotiations and prices
The prices of some solutions are unbelievable. With our capacity and the need for a redundant solution, we are talking about amounts that have 7 zeros. Sometimes DDoS protection can be bought as a service, but at our capacity it’s 6 zeros a year.
Our solution cost us a lot of work, which we do not count (it would be thousands of hours) in development, but otherwise the hardware we purchased for millions of crowns and the performance for filtering and detection is many times higher than the tested solution. So now you actually know how much the whole DDoS protection cost us…
Solution selection and preparation
The whole process of selecting, negotiating with suppliers and testing took us several months. In the end, we made the right choice, and with hindsight we know we made the right choice.
Unfortunately, we can’t reveal too many details about our solution, at least for now, because that could make the attackers’ job easier.
Deployment unplanned
Due to thorough testing, we were able to deploy the basic solution almost immediately. In the end it was necessary to do so, because there was a strong attack going on that could not be dealt with in any other way. So last October we simply had to disconnect the cables and plug everything into the test solution. There was no other option. Trial by fire. That’s what it’s called, but it worked and today we know it was a good decision.
An endless story
The deployment of the test solution was just the beginning. Gradually we had to debug the mistakes. We had to change settings and gradually increase capacity and power. Today we are at the stage where we have not changed anything in the last 5 weeks and everything is working as it should. We are just tracking the number of attacks, their intensity and collecting valuable data that may not only serve us in the future.
We’re not resting on our laurels. We are working on further improvements and performance enhancements. The attackers are resourceful, so we must not fall asleep.
How is it now?
The whole solution is strong Linux servers and strong switches
Everything is built on powerful servers that have either a lot of threads (40) or very high frequency (3.5 GHz or more) processors, depending on what the server is supposed to do. Sometimes you need to serve a lot of requests at the same time and sometimes less, but as quickly as possible.
Perfect detection
Everything is based on perfect network traffic detection. We have 4 online probes that monitor all network traffic online (in real time) and this is immediately evaluated.
Each probe can currently detect 10 Gbps and so we can easily measure and evaluate 40 Gbps. If the attack was stronger, it wouldn’t matter because it’s perfectly sufficient for detection and evaluation and we just won’t get as accurate results.
Every 5 seconds, the individual probes evaluate whether it is an attack, and so we have a theoretical delay of 1.3 seconds before we detect an attack.
Perfect overview of what’s happening on the network
In addition to the above mentioned probes, we have 4 more probes in the network that work on the sFlow principle and monitor and record all traffic in the network for more advanced evaluation and processing.
Each of the probes can process up to tens of Gbps, so we do not have a capacity problem even during the strongest attacks.
These probes are then used to process various statistics and search for traffic on our network. They are not used directly for processing attacks, but for analysis, because they are accurate and record history, but are delayed by 30 seconds.
Perfect filtration
We have several filters on each optical route to our datacentre, to which faulty traffic is redirected in the event of an attack. If there is an attack, the problematic (attacked) traffic is redirected in real time to filters and all traffic is filtered or limited in some way according to certain criteria.
We can filter, we can restrict, we can do things. Everything is dynamic, depending on what is happening at the time.
The filtering is extremely computationally intensive, so the filtering is performed by several parallel filters.
Each of the filters has several 10 Gbps cards in it.
Additional filtration stages
We have additional filters behind the border routers that we use to clean up the faulty flow in detail. These filters are used for cleaning afterwards so that we do not let “mess” into the network.
Filter performance can be scaled by adding additional filters either in parallel, where they share the load, or in series, where each filter specializes in filtering certain traffic parameters.
Filtering outgoing attacks
To improve the whole system, we also added filtering of outgoing flows. This prevented weak attacks from going out to other targets on the Internet around the world and coming back to us amplified in response.
Routing
We’ve made a lot of adjustments to the routing. We can BGP blackholing and we have also set up selective BGP blackholing. The system automatically changes the BGP routing settings based on what the attack is and how it is configured for a similar attack. In the event of a strong attack, we can completely delete the IP address in question from routing tables around the world or just outside the networks of our peering partners and their direct peering partners. We use this only rarely and only for the stronger attacks. Otherwise, we always try to filter.
Replacement of backbone elements
We replaced the backbone routers and switches with other – more powerful ones. Today we have huge reserves and thus an advantage for the future. We can expand in peace and we don’t have to change anything.
Strengthening the backbone infrastructure
We have replaced our entire backbone network and it is fully fibre optic and you can see it at the Open Days we are planning for early October 2015.
All network elements in our datacenter will be ready for 20 Gbps. So each switch will be connected to 2 x 10 Gbps. The backbone switches are fully 10 gig (all ports are 10 Gbps or 40 Gbps).
Strengthening connectivity
We have strengthened the connectivity and currently we have 70 Gbps connections that we can not only unscrew, but also mainly filter out the faulty flow. In the next few days we will add another 10 Gbps link and then we want to add one more 10 Gbps route to Prague.
Too many settings
The system is evolving and we are working on improvements. There are a lot of settings and a lot of different “tweaks” that we use. Piles of scripts, piles of settings… It’s really complicated. From detection to filtering, it also controls routing and switches and network flow.
How did it turn out…
The last problem with DDoS attacks that a client may have observed was on the 29th. December 2014. Since then, we have had no global wavering.
We have had 100% availability for the last 6 months and in the meantime we have changed the entire backbone network. So even the replacement of the backbone routers was successful without any failures.
We are constantly looking for improvements and detecting other threats. On the one hand, there are new threats, and on the other hand, there are old threats that we have not yet detected and so have not addressed. We’re improving all that.
Some interesting facts about DDoS at WEDOS
Some numbers and interesting facts about how it was in our country…
The most frequent attacks were on 2 of our web hosting servers and then on 1 VPS. The number of attacks on each of these “winners” reached 4,000 for the period under review, and the winner has even 4,773 attacks.
The strongest attacks were on our website www.wedos.cz. The strongest reached more than 30 Gbps and more than 4 Mpps.
A little off topic…
This year we are planning a few new things. What kind?
Paid DDoS protection
We want to offer clients paid DDoS protection, where we will offer traffic filtering up to several Gbps and protection against attacks in different capacities. We are still considering all options and testing. We wonder if there will be any interest. It will be a more expensive service, but it will be for demanding clients who cannot afford an outage.
Starting a brand new mailserver
We will be launching a brand new mailserver in the next few days. We’ll gradually move all the clients there. We will write our own article about the change and will keep you informed.
We promise more stability of the mail solution than before.
At the same time, we also offer the possibility to order the mails themselves to VPS or web hosting.
New optical route
We would like to have one more completely separate and independent optical route to Prague.
Start of construction of a brand new datacentre
A few days ago, we purchased a plot of land and this year we want to start building a datacentre, which has been in preparation for a long time. In the future, we want to have 2 datacentres that will together form one cloud and we will start offering cloud services.
We expect the highest possible level of stability and security from the second datacenter.
Open Days 2015
In October we are planning Open Days in Hluboká nad Vltavou. We will bring you the details soon.
Several technical innovations in the offer
We are preparing several new products. For example, we currently launched VPS 100% SSD Profi.
Multihosting or managed VPS
We have been promising proper multihosting for a long time. It’s already in the works. We’ve been delayed by DDoS attacks, so there hasn’t been time…
New domain extensions and custom ICANN registrations
Now we are preparing the registration of Polish domains and we want to offer other endings. We are also working on our own accreditation for generic domains.
New markets
Later this year we would like to offer our services for example in Poland and maybe in some other country…