We have been preparing IDS/IPS protection for you for several months. In March, we launched it into live operation. Today we’re going to introduce her to you.
We would like to start by apologising for any complications we may have caused some clients in recent weeks. As an explanation, we send the following article.
When we were the target of strong DDoS attacks (tens of Gbps) in the second half of 2014, the only solution was to build our own protection or rely on our connectivity providers. None of the contractors had experience with such strong attacks, so we only had the option of doing everything ourselves. We decided to mobilize our forces, invest millions in rebuilding the network infrastructure and purchasing new elements. By the beginning of 2015, we knew we had done the right thing.
Thousands of hours of work and millions of dollars of investment have paid off. Even when DDoS attacks have increased to the point where our connectivity providers could no longer help us, our protection can stop it all. Of course, it took many more months before we tuned it to the state it is today. In other words, we no longer closely monitor attacks below 10 Gbps, we just look at the log entry. In most cases, no unusual traffic reaches our servers and does not compromise the service in any way.
Just to give you an idea, we have had over 210,000 DDoS attacks since the end of October 2014. That’s an average of over 12,000 attacks per month for 17 months, an average of over 400 per day. Every week there are attacks exceeding 10 Gbps and practically every day attacks exceeding 5 Gbps.
Our business success and growth began to interest (besides the attackers) various suppliers. Due to the number of hosted websites, we are a very interesting customer for security companies from all over the world, who have started to approach us with offers of various cooperation. For some it’s about collecting data, for some it’s about selling their own solution, and for some it’s about both. Rebuilding and improving our network infrastructure, testing exclusive software and hardware opened up new possibilities. We realized that we can offer our customers much more these days. The most advanced protection we can offer – IDS/IPS.
What is IDS/IPS protection
IPS (Intrusion Prevention Systems) and IDS (Intrusion Detection Systems) is protection that monitors absolutely all traffic between the server and the client. It can monitor it in real time, evaluate it completely and, if necessary, block anything online. Both ways, of course. It can detect and prevent attacks, but it also works as a prevention. The scope and capabilities of IDS/IPS are very comprehensive. For example, it can delete infected emails, block your website from being overloaded, and also stop an attempt to exploit a zero-day exploit even if it hasn’t been reported yet. It looks for specific strings for different types of attacks that do not occur in normal operation. If it detects them, it acts immediately.
Why we all need her
Nowadays, the number of open source content management systems and e-shops has increased significantly. Simply use our app installer and in two clicks you have your own magazine, discussion forum or e-shop. You don’t need to have any programming knowledge. You don’t even need a webmaster to take care of everything. And this is where the problem arises. All these open source solutions need regular maintenance, especially updates. Attackers are looking for any loophole to get malicious code onto your site. It’s not done by humans, it’s done by their robots that can go through hundreds of pages per second.
Moreover, every open source solution has a serious bug once in a while. Often you only have a day, or at worst a few hours, to update. If you don’t make it in time, your website will be visited by a robot that exploits the security hole and uploads an inconspicuous backdoor through which the attacker can control the website. Even after the update, you are no longer safe.
So just go on vacation for a few days and you can have your website hacked. We deal with such cases on a regular basis.
But it doesn’t have to be an open source solution. Every programmer has to deal with untreated inputs. Keeping track of everything is not easy.
Our new IDS/IPS protection is designed to protect you all. It can’t do everything, but it addresses most known security vulnerabilities.
What our IDS/IPS protection can do so far
We are currently tuning the IDS/IPS protection. It’s not easy, but we believe that the large (financial and time) investment will pay off.
It currently protects your services from:
- Natural-looking DoS that our DDoS protection cannot detect (for example, XML-RPC abuse).
- SQLi attacks
- Brute force attacks on login forms.
- Robots that deliberately overload your site.
- Robots that insert malicious code (XSS) into comments
- Viruses in emails and even regular HTML (FTP) traffic (but we are still tuning a suitable traffic model).
- Vulnerabilities in well-known open source CMS and e-shops.
- Zero-day exploit attacks (known and potential).
- By scanning various applications.
At the same time, our servers are protected against the exploitation of various errors in the editorial systems that can result in server overload or, for example, exhaustion of RAM or CPU power.
In addition to inbound attacks, it also monitors and blocks outbound attacks. We can block outbound attacks and minimize potential damage in the event of a website attack. Usually this happens automatically, so that the IDS/IPS protection automatically blocks (resets the ongoing connection) the connection between the attacker and the target.
We also have a more detailed overview of what is happening in the network, which opens the way for us to improve our services in the future. For example, monitoring services for you, above the server level.
IDS/IPS protection is free for clients and you don’t have to do anything to activate it
The management of WEDOS Internet, a.s. has decided for the time being that the new IDS/IPS protection is free for all our customers with the NoLimit web hosting service. It is automatically activated above the physical server level, so you don’t have to worry about anything. We believe it will improve our services and save you a lot of trouble. We are constantly working to improve our services and believe that IDS/IPS protection is the next big step forward.
IDS/IPS protection at WEDOS
We currently have a commercial solution set up and in use, which we want to supplement with open source. The total cost of the deployment is estimated at several million crowns.
The testing process has been ongoing since the end of summer 2015. Preparations for deployment were continuously made from January to March this year. It was not easy and involved a lot of adjustments to our network topology. We’ll do some more editing in April (and probably finish some of it in May). We launched everything in March. We will write about all the things we had to deal with during the implementation next time.
In the next article we will describe how the deployment of IDS/IPS took place in our country. It wasn’t simple or easy. In some cases, this has caused minor complications for our clients. Again, we apologize for these complications, but we believe that the benefits of IDS/IPS protection will be so great that it will be worthwhile.
Some basic numbers
Our network hosts the most websites in the Czech Republic. At the same time, we host the most virtual servers and all this puts a strain on the security of our network and the applications on our network. When we started “flirting” with IDS/IPS and I started the first tests, we discovered some incredible facts.
For example, every hour, our network received:
- an average of over 37,000 password cracking attacks in WordPress, which is over 10 attempts per second
- roughly the same number of attempts were made to crack other passwords (emails or other content management systems)
- an average of over 1,000,000 different security issues, which is over 279 attempts per second
- of which over 183,000 had a critical level, i.e. it was not a warning or low level, which is about 51 attempts per second, which are dangerous for clients
- from about 100 IP ranges from all over the world came about 73% of the malicious traffic and so after limiting and tightening control against these ranges we reduced the number of attacks to about a quarter
We will provide further figures next time.
For interest from behind the scenes
Some of our webservers have been under constant (or frequently recurring) attack to overload a particular webserver. How did such an attack take place? All possible domains hosted on a specific server (according to the target IP) were called from different IP addresses on specific servers and certain uncached URLs of known content management systems (various administrations, xmrpc2, etc.) were called on them.
The attack was carried out from different IP addresses and ranges, with each IP address so that the number of accesses from one IP (or one range) was not noticeable. Moreover, these were not accesses to a single domain, but to different domains. So everything was spread out over time and from different sources and to different targets so that it didn’t attract any attention. The result was that on some web servers we had over 50 accesses per second that were to uncached pages and had access (and write) to the database.
This was not a classic DDoS attack, but a newer and less noticeable and harder to catch form of attack, where this unnecessary load caused complications on servers and slowed down some websites. IDS/IPS protection helped detect and filter this (new to us) form of attack.
What next with IDS/IPS
We currently have IDS/IPS protection deployed on all web hosts. We want to extend it with other improvements and we want to add some outputs (statistics) to the customer administration. However, this is likely to be an extra cost service. This is because it involves logging a large amount of data.
If we combine IDS/IPS protection with DDoS protection, your sites and your clients’ sites will be very well protected.
We are still discussing the deployment of IDS/IPS on VPS and do not know whether we will offer it or not and if so, under what conditions. In the case of VPS it is the case that some clients are not interested in this protection and some are. We’ll see how we work it out technically and commercially. You can write us your opinion in the discussion below this article. We’d be happy to.
And what can you look forward to next?
Next time we’ll write about what complications we had to deal with during deployment. And finally, we may reveal how we have everything wired and what it all runs on.
To entice you, we must also write about the construction of our new (second) datacenter, which will be exceptional not only in terms of physical security, but also in terms of ecological and economical operation, because the servers will be cooled in an oil bath. Once we launch the second datacenter, we want to offer services with a guarantee of high availability, where it will be possible to have data in both datacenters at the same time, in both locations protected by DDoS protection and IDS/IPS. In case of an outage of one site, your website would run automatically and without any outage from the other site. More on that next time. The development of all the new products and the construction of the second datacentre are now keeping us so busy that we write little about what is happening here.