We’ve been going through hell for the last 3 months. A challenging period that has taken us very far. At one point we thought we were going to have to change our name from WEDOS to “WE DDoS”.
WE DDoS
It was (and still is) a very challenging period for us. Since the second half of July, DDoS attacks on us have been getting stronger and stronger. Let us summarize some basic information concerning this period.
We used to survive attacks without any problems. They weren’t as strong, they weren’t as long, and they weren’t as repetitive. We have always been able to do this because we had enough spare capacity (connectivity). In the last 3 months the attacks were so strong that even the 7 times reserve of our network was sometimes not enough.
For the last 3 months, we have done almost nothing other than fending off a strong attack or looking for solutions to handle the situation better next time, and when it was at its worst, we spent our time explaining the situation to customers and fans on social media.
We’re moving forward gradually. Not everything is ready yet. Nothing is in the final version yet, but we already have the first positive results, where some attacks do not reach our network at all. You can see it in the charts in the last few days when everything is absolutely fine.
It is clear to us that once we publish this article, we will probably be the target of further strong attacks for everyone to try. Or whoever has been attacking us so far will want to test our defenses. We still have a lot of work to do. And in the following lines we will describe the procedures that explain the whole issue.
Understand – it doesn’t happen right away. It’s not. I’m sorry.
All the time we were not only solving the situation, but mainly figuring out how to do it better in the future. We have dealt with manufacturers, we have dealt with suppliers, we have dealt with “advisors” in this area.
It cost us a lot of effort, it cost us a lot of money and it cost us a lot of time. All the time we did almost nothing else (meaning the group of people involved in development). We tried, we tested and finally bought some things and we will buy more. It cost us a lot of time. It’s thousands of hours of human labor that could have been spent on something else…
This is what the attackers were and are all about. To damage us in the public eye, to hinder our development, make us spend a lot of money. As always, we faced the situation head on. When there was a problem, we admitted it and described it. We’ve only missed it in the last 10 days when we were making big adjustments, and we put all our efforts into making changes as quickly as possible.
The attacks on us were sometimes so strong that they were sometimes among the biggest in the world. You can’t build protection against such an attack in 5 minutes and realistically nobody in the Czech Republic has it and nobody can provide it to you on the spot.
Moreover, the situation needs a thorough analysis, and there is also a problem with the particular structure of our network, and if you want to have everything backed up, you just can’t figure it out on the fly. Without a backup solution, it doesn’t make sense for us. On the one hand, you are asking for a problem and on the other hand, filtering 1 line out of 3 is not enough. The offense in that case will come through second and third.
Real impact
We apologize to all customers who have been affected by the DDoS attacks. On the other hand, it is necessary to write that within 3 months we have experienced 2 major complications that affected a larger part of the clients and then about 2 smaller ones that affected only a smaller part of the clients. We have had more complications when one of our providers was down due to attacks (on us) and at the same time we were under attack (and thus lack of capacity). One time it was complicated for 5 minutes and the second time for about 20. Two minor problems we had were when there was a short-term congestion of lines to the VPS or webhosting and before filtering, there was a degraded response.
We admit that it had a much greater impact on our website and customer administration, where the strongest attacks were directed. The situation there was more complicated in the short term, but again it did not have a direct impact on customer servers and services provided to clients.
Often there was more written about it than the actual impact. Many customers monitor their services via ping or monitoring services that use ping. At the time of the attack, we were blocking pings very often, because we had situations when pings from all over the world were coming into our network and the data flow reached several Gbps. Thus, monitoring based on pinging the server usually did not show correct results. It could have looked like an outage for a few hours and in reality there was none and everything was up and running for the clients. Only for the duration of the attack was pinging banned to our entire network.
Based on this erroneous information from the monitoring, very often clients would call us to say that something was wrong and sometimes it was difficult to explain that everything was working and only their monitoring showed something else.
At the same time, it also happened very often that someone posted on a social networking site that we were under attack again, and that was not actually the case. Several times it happened that there was an attack on a customer’s VPS (maybe even several Gbps) and this client had his VPS unavailable and according to this he concluded that the attack was on us and that we were the “culprit”. When he then posted it somewhere on social media, it then spread itself further and it seemed to be about all of our services. There was a panic about it and it was about the unavailability of one particular VPS. This has happened, unfortunately, repeatedly.
On social networks, many different anonymous profiles or profiles that have been set up by someone to spread information about complications (even though such profiles look very trustworthy) and, most importantly, they are not our customers at all, although they often pretend to be.
Occasionally some graphs showed nonsensical numbers and so it looked like a drop in data flow or a nonsensically high flow. This was due to the fact that we made various interventions in the network and added filters, sensors and other devices, and at the same time we changed routing or created waves. Thus, the graphs showed one way and the other way. However, the graphs are only indicative.
The situation in the last 3 months has certainly had a negative impact on some of our clients. There is no doubt about it. However, the real impact on services has been considerably less than many of the comments make it seem. Coincidentally, it is true that most often there are people who have a VPS that has been repeatedly attacked and so it has had a negative effect only on that particular VPS, and yet it often looks like the entire WEDOS has had problems according to the posts of the owners of those VPS. It wasn’t.
Do not take the above words as a downplaying of the situation, but rather an explanation. The whole situation was and is a top priority for us.
A few interesting facts
- Forbidden pings or don’t trust monitoring
As already mentioned above, monitors that only use ping cannot be trusted. We have very often limited or banned it altogether. The strongest flow via “pings” to us was captured at about 3.2 Gbps per IP address at our site.
- Availability
In each of the individual months we have always had the availability of our network as guaranteed (maximum 5 minutes of downtime per month). For some services, accessibility from some locations has been impaired for some time. From our perspective (from our network or from domestic sites) all servers had fine availability. According to the monitoring, the service may have been unavailable (in terms of ping) or there may have been a slow response from some locations. What often happened was that from some routes (where the attack was coming from) the response was very poor, because the routes towards us were simply and simply (literally) clogged.
- Related issues with external sources
There were several times when we were under attack, some protocols were disabled or some connectivity directions were literally clogged, but everything worked for us. Some customers have reported that their sites are slow to load. Yes, this was caused by, for example, generating a page full of external links and some of them were unavailable. If such a link was inserted into the site in an inappropriate way, the front page was not loaded and the external source was waited for.
- The attack from our VPS on us
It is also interesting to note that we have also intercepted attacks on us from our customers’ VPS. These VPS are probably compromised and are part of some botnets that are used for similar DDoS. So protection is not enough from the outside, but we have an “enemy” on the inside as well.
- The most vociferous often find themselves under attack…
As written above, many of the most active contributors on social media (writing about the attack on WEDOS) had their VPS under attack and often mistakenly passed it off as a problem with WEDOS or an attack on WEDOS as a whole. Often this was not the case, but this only started to emerge in the last few days when we started to do absolutely detailed analysis of network traffic. We have notified some clients and we will notify some more. We will then handle this automatically.
- Most often game servers or teamspeak
The most frequent problems were noticed by owners of game servers (on VPS) and users who have teamspeak. Yes, it’s logical because these are services that are very sensitive to degraded response. On the other hand, we have seen some infighting between these servers, which seem to settle accounts among themselves and then send each other traffic that is not directly a DDoS attack, but just an attempt to overwhelm the “opponent’s” virtual server with requests. Such requests are often from one place (another game server). Again, these situations were often passed off as a DDoS attack against us and WEDOS problems. In reality, this was not the case and such situations did not have a negative impact on other clients. However, what was often repeated was that several players from the same clan would post on social media afterwards and it would look much worse than it actually was.
- The strongest attacks on our network
The strongest attacks we had were around 30 Gbps and 6 million packets per second. The most frequent attacks (in terms of strong ones) were 13-19 Gbps and around 4Mpps. It always took a few tens of seconds. It started suddenly (within 1-5 seconds) and after some time it stopped and dropped to about half (the protection measures of the providers “above us” worked).
- The strongest attack
The strongest single attacks we recorded were 4.6 Gbps (one form of attack per IP) and over 2 million packets per second. However, most of the attacks were combined, with several different forms of attacks on a single IP address (at the same time). So the resulting flow per IP address often exceeded 10 or more Gbps.
In the last week alone, we’ve had dozens of attacks that were stronger than 3 Gbps and dozens of attacks that were more than a million packets per second.
- The longest attack
The longest attack lasted over 7 hours.
- Most attacks in one day
The most attacks in one day we experienced on Sunday 12. October, when over 1,100 different attacks took place within 3 hours.
In the last 5 days, over 2,100 different attacks have been directed at us.
- Some of the most powerful attacks in the world on a particular day
In the world, the attacks are stronger. You can find attacks of 50, 80 or more Gbps, but even so, attacks on us were among the top 10 largest in the world in a single day. I’m sure there’s nothing to brag about… but just out of interest.
- Sources of attacks
Of course we have statistics. It was from all over the world, but we also saw one targeted and powerful attack from a datacenter in Frankfurt, where several servers were attacking us simultaneously and hundreds of thousands of other computers from all over the world were “bushwhacking” so that it would not be obvious that the source of the attack was targeted in this way.
- A bit of statistics on the attacks
The strongest attacks were UDP packets and ICMP. These are quite prevalent among the biggest attacks. Furthermore, various forms of TPC attacks and TCP SYNC attacks are prevalent. Other attacks are completely negligible. The attacks were directed to different IP addresses. For example, in the last week there were 241 IP addresses. Mostly our website and our company stuff. Much less often on web hosts and VPS. Very often, short attacks are repeated, which are a “sort of feeling out” of the network, and when they pass, longer and stronger attacks soon follow (tens of minutes long). Attacks most often come in the afternoon or evening rush hours (around 13:08-14:00, 15:15-17:00 and 18:00-22:00).
- Over 700,000 site visitors per second
If you want that kind of traffic too, you just need to be successful like WEDOS and someone will provide it for you. He’ll order a DDoS attack and then you’ll have it. Yes, our website received over 700,000 requests (from 700,000 different IP addresses) per second during one of the attacks . Unfortunately, our webserver is not built for this, so of course it rejected the requests and our site was unavailable.
- “Dumping” both suppliers
During that period, our suppliers were also inconvenienced. Due to the severe attacks, they experienced connectivity outages that resulted in problems for their other customers. We apologize for the complications.
- Strange testing
We also find it strange that we had a period of very strong attacks. Subsequently, we were contacted by several vendors and service providers for DDoS protection. We tested some products and during these tests there was almost absolute calm. Just for the sake of interest, we should add that for example before testing we detected attacks of 20 Gbps or more and during testing (about a month of different products) the strongest attack was 3.5 Gbps. After the end of the tests and the return of the tested machines, we had about a week (when we discussed possible terms of cooperation) and then again came more attacks. Even stronger than before. It seems like it’s in a series – “you need it”, “try it”, “see – it works”, buy it”, “are you hesitant?”, “so here are more attacks to let you know you absolutely need it”. It looks conspiratorial, of course, and it’s probably just a coincidence, but… Perhaps we can explain it by the fact that the attacker was watching us and knew that we had protection in place and that the attacks would be completely useless. There are more possible variants…
- Targeted at WEDOS, really targeted at us
The attacks, especially the strongest ones, have targeted the very essence of WEDOS. It was attacks on our servers, where our website runs, our administration. It was always an attack on us. In addition to attacks on the web, we have had attacks on our corporate firewall where someone tried to send over 10Gbps and thus completely cut us off from connecting to the world. It was certainly not an accident, but a targeted attack on WEDOS.
- Someone’s watching us
It is interesting to note that the attacks were directly related to what we were writing or doing. When we had an interesting promotional event, someone launched an attack on our website. When we put an interesting post on social media, within minutes someone was making attacks. As we were adjusting our network and going purposefully over the backup line (a few dozen minutes), someone launched an attack on us. That’s an awful lot of coincidences that certainly already disprove the assumption of random behavior and random attacks. Nobody believes that anymore.
There was no attack for 3 days, and the moment we were rebuilding the network and we were in limited traffic and something was written about it publicly somewhere, someone sent an attack right away (in a few tens of minutes)…
- Click and go
DDoS attacks usually started strong, at one point in time. All at once. The onset was within seconds (units). The duration varies, the strength of the attacks varies. First some form of attack was tried, then another, other targets, and so on. Most of the strong attacks came quite regularly. We’ve joked that we can set our watches accordingly. The flows ended gradually. A strong attack for example in exactly one hour (he was simply paid for one hour) and he had reverberations for several hours afterwards.
- One IP at a time…
We’ve seen attacks where someone would send an attack for 5 minutes on one IP address and another 5 minutes on another IP address, and they would go number by number like that and go through the entire range that we use for our “internal” stuff (meaning our website and client interfaces). It was a DDoS scan of our network…
- When it didn’t work here, they tried elsewhere…
When our network resisted, the attacks were moved against our servers in another location.
- Never-used IP addresses also targeted by attacks
The big interesting thing is that we have “borrowed” IP addresses for VPS from one provider, because we are running out of them. These IP addresses have never been used and have never been in use. They’re just grafted on (pointing towards us) and there’s no traffic on them. Nothing happened. Only the provider in question registered them in the RIPE databases at WEDOS and within a few days very strong attacks started coming in on these IP addresses… Just because these IP addresses were registered to us…
- The perpetrator? Unknown…
There’s no point in even looking for the attacker, the source of the attack. That’s unrealistic. These are illegal networks of unsecured and hacked PCs from all over the world…
- Why?
Success is unforgivable. There’s no other reason. It is targeted against us, targeted against the essence of our business. Yes, we’re doing well. We have (despite the attacks) almost 70% market share (in terms of growth) and that is unimaginable and unacceptable to many others. So this is what he’s trying to do. Previously it was gossip (apparently to no effect), then it was denunciations to all kinds of authorities (no doors were broken off with officials, but no collusion was found), and now it is simply this form of attacks – virtual terrorism, one could say.
- I’m sure it wasn’t free
Such attacks are certainly not free. It must really bother someone to go to all this trouble to torment us like this. We must bother someone so much that they are willing to spend so much time planning attacks on us…. We have to mind so much that someone is willing to pay for it. Success is unforgivable.
What did we do?
We have devised a completely new wiring scheme for our network.
In real terms, we strengthened the network by another 10 Gbps (directly in Prague).
We have tested many different solutions.
We have dealt with many suppliers.
We made a lot of changes to our network. We have so far added a total of 7 physical devices to our network to detect and deter DDoS attacks.
We have already had the first positive feedback and impacts. We’ve picked up a few strong attacks that none of our customers have noticed. Neither our monitoring, nor any of our people, have detected them. We just saw data flows in our internal statistics that didn’t make it into our network at all.
We are still testing other protection options (we are testing HW, SW and some solutions sold as a service).
We are currently fine-tuning all the rules that are currently in use. It’s quite a science, so we still have a lot of things to debug, because you can’t simulate it in a test environment.
What else are we gonna do?
We will add more routes and connectivity to the network. We will add another 10 Gbps route to Hluboká. In Prague we will add at least 2 or more different links (each 10 Gbps) to other providers.
We will add at least 2 more devices as fitters very soon, but it will probably be 3 or more.
We’ll replace our backbone routers with different ones. Much stronger and more powerful. This is not because the existing ones have run into any performance problem, but as a precaution.
We will use a completely different network management. We will switch to SDN – software-defined networking, where we will control our switches and routers from a PC using commands.
What do we use?
We don’t reveal what devices we use, and we don’t reveal the wiring diagram, and we don’t reveal the attack graphs (before our filters). Why? It’s very simple. We’d make the job easier for the other attackers.
We won’t even say what attack capacity we can handle now. We won’t write down any more details.
When everything is ready, we believe that everything will work very well and very reliably and at the same time we have one very crucial thing. In the event of a capacity shortage, we will be able to increase the filtering capacity within a few days.
It’s a solution that doesn’t come for free, and it took a lot of effort, time and money to select and prepare this solution. We don’t want to make it easy for anyone in the future.
Don’t ask us. We’re not really gonna say that. In this case, secrecy is simply in order. Thank you for your understanding.
Finally, we apologize once again
In addition to an apology, we’d like to add one more request. Have a little more patience with us. The whole solution is not yet in final implementation in terms of hardware. That’s gonna take another 2-3 weeks. Likewise, it will take us a while to fine-tune all the rules. We’ve now set up basic protection against the attacks we’ve seen so far, and we’ll definitely be adding more. We will have to tweak and adjust the rules in different ways. We’re working on that with the manufacturer now. It’s not easy. We have quite large data streams, diverse services and so the setup is slowly becoming a separate discipline.
At one point we thought we were seriously going to rename ourselves to WEDDoS or “WE DDoS” etc… We didn’t give up. And that’s why we believe all of our moves will move us very far ahead and ready for further growth in our services. We believe that the protection that we are gradually deploying will be very high quality and stable. We don’t know of anyone (in the country) who is prepared to filter tens of Gbps of traffic online against DDoS attacks. This is the thing that sets us apart from the competition and puts us very, very far ahead of other companies.
We are glad that you keep us in your good graces, thanks to you we can grow and improve our services. Thank you. We really appreciate it and this is what drives us forward.
Again, what doesn’t kill us makes us stronger. This will definitely make us stronger. It’ll make him a lot stronger.
If you want to discuss with us, please be factual and sign and indicate what services you have with us. We don’t want anonymous, arrogant and completely unnecessary discussion here that is not on topic. Thank you for your understanding.
Refill at 17:00
This article was published at 15:37 and at 16:13, apparently as soon as the attacker finished reading, we were subjected to a sharp stress test. For about 12 minutes, about 8.5Gbps of data (and about 1.5 million packets per second) was routed to us. You don’t see it anywhere on the chart. None of the customers noticed and no one noticed anything. The protection worked. However, we are not done yet. We have to debug and it’s going to take some time.
Footnote… and a little off topic…
It may be unrelated, but we’ll post it here. As part of the (anti)WEDOS campaigns against us recently, we have been labelled several times as “terrible crooks” for not fulfilling our duty to the commercial register and for not putting information about our management there. Yes, it’s true. We’ve already fixed it (and it will surely show up in time). On the other hand, people who are not our shareholders and often not even our customers bring it up. Basically, they don’t care about our economy… but they have nothing negative to pull on us anymore, so they try this too. They always remember to write that it is illegal and unlawful and that we are not following the law. It’s about making us look as bad as possible in the eyes of the readers…
We are aware of this so we have corrected it and will keep an eye on it next time. Our company is growing, generating an operating profit (the accountants have not yet) and we are constantly investing in further development. We are constantly acquiring new servers, now for example DDoS protection and now we are already investing in building another datacenter. We are always moving forward and accounting profit is not a benchmark for us. The benchmark for us at the moment is the development of society and the development further and further into the future. We just want growth and happy customers.
End of the exercise.