At the beginning of the year, we significantly strengthened the parameters of the NoLimit service, especially in terms of computing power. This was especially appreciated by customers using content management systems with larger databases and customers whose websites are visited by a large number of visitors on a spur-of-the-moment basis. And it shows. The number of service cancellations due to “dissatisfaction with performance/speed” dropped almost 4 times. But it’s not just about increasing the parameters. Behind it are changes and advances that most customers are unaware of.
To give you an idea of what NoLimit can currently handle. Check out the stats on the number of requests for the busiest hostings on Monday 26. 09. 2022. This is the traffic that reached the destination server and it had to return/generate the page. Depending on how the customer has caching set up, some of it is handled by the proxy server (these requests are not here) and there are also no accesses blocked by WEDOS Global Protection. The traffic that these customers would have to deal with on their own server would in some cases be several times greater.
These customers pay the regular NoLimit or NoLimit Extra, which we offer for CZK 68.80 or CZK 135.80 per month excluding VAT. Websites on the LowCostplan of customers who know how to optimize their website, manage traffic in tens of thousands per day (around 100 thousand requests).
NoLimit is a universal web hosting, optimized for the widest possible use. It can handle both popular solutions (WordPress, Prestashop, etc.) and customized solutions. Do you think that such sites could work on it only if they had above-standard parameters?
The truth is that NoLimit is backed by a robust infrastructure, branded quality hardware, and a team of people who have years of experience running over hundreds of thousands of hosting services. There is no one else like that on the Czech market, nor has there ever been. We became the number one hosting company in the Czech Republic in 2013 and have been growing ever since. We don’t kill anything, on the contrary we improve everything and make it even more robust. And when you add years of experience…
WEDOS Global Protection for free
In recent months, our top priority has been to build the WEDOS Global decentralised network. This not only greatly increases the speed of all DNS queries worldwide, but also allows us to build other services on top of it, such as cyber-attack protection, CDN (cached content storage at a given location), VPNs, and so on.
We need cyber protection at every point because the latest attacks on us and our customers have already exceeded the limit of what can be realistically protected in the Czech Republic. First of all, 300 Gbps of traffic is already known, but most importantly, such massive attacks go through a number of ISPs that do not have a network built for this. Computers, mobile phones, televisions, refrigerators, anything connected to the Internet, are also infected inside Czech networks, and there is a lot of it. The last attack generated traffic in the lower tens of Gbps and it will get worse.
In September, WEDOS Global consisted of a total of 15 points:
- Europe
- 🇳🇱 Amsterdam
- 🇪🇸 Barcelona
- 🇨🇭 Zurich
- 🇫🇮 Helsinki
- 🇨🇿 Hluboká nad Vltavou
- 🇬🇧 London
- 🇪🇸 Madrid
- 🇫🇷 Paris
- 🇨🇿 Prague
- 🇸🇪 Stockholm
- 🇦🇹 Vienna
- 🇵🇱 Warsaw
- Asia
- 🇭🇰 Hong Kong
- 🇸🇬 Singapore
- 🇯🇵 Tokyo
At each point there is at least 1 HPE Moonshot system with 45 physical servers and 80 Gbps connectivity (we will gradually connect to local ISPs and the connectivity will increase to 120 – 160 Gbps). That’s 1200 Gbps (1800 – 2400 Gbps), which we can filter and moreover locally. Only clean traffic gets into our datacenters.
The plan is to build a total of 50 such points in the first phase.
WEDOS Global Protection is already being finalised. It is currently used by about 600 websites of our customers that are regularly under strong attacks or have high traffic overloading their hosting. We’ll show you what kind of traffic it is in the chart from Monday 26. 09. 2022. Note, if you want to compare with the previous table, there are also (sub)domains. There is also a version with www. and without (for each of them the requirements are separate).
Without WEDOS Global Protection, these sites (including ours) would not be attacked. If this were to happen again, none of the competitors would want to host them on a shared webhost. Competitors often solve this by disconnecting the site from foreign connectivity, which helps temporarily. But it won’t please the owner. The permanent solution is to pay for third party filtering or … switch to WEDOS 🙂
With us, such protection is free of charge. We are currently testing it on customers who are under attack, but we will gradually deploy it to everyone for free. There will be universal and tried and tested filters.
The cost of this protection? The list prices of hardware are over 100M CZK, the location of servers, connectivity and electricity are currently hundreds of thousands CZK per month, but gradually we will move over a million CZK per month.
Below you can see some graphs from real attacks that went through WEDOS Global. Not everywhere is set to active protection (something is still activated manually), so in a few minutes before the activation is done, the attack passes and creates hundreds of thousands of errors. This is still a test run 🙂
Superior infrastructure protection against DDoS attacks
Nobody in the Czech Republic seems to have as much practical experience with DDoS attacks as we do. We hold the record for both 2021 and 2022. We are not aware of anyone who has ever dealt with such strong attacks in the Czech Republic. And we are in contact with people who provide connectivity in the Czech Republic and have an overview, as well as with the National Office for Cyber and Information Security (NUCS), to which all significant attacks must be reported.
Our job is even more complicated because we don’t protect 1 service where you know exactly what traffic you want and what you don’t. Our customers run all sorts of things, using shared web hosting, VPS, dedicated servers, cloud, etc. They have traffic from all over the world, projects connected to various APIs that they need to communicate with quickly, and third-party services.
Designing DDoS protection for all this is quite a challenge. In order to do this, we work with a daily log data volume of 1042.45 GB (this is normal traffic when there are no massive attacks). We automatically create filters from these and look for new types of attacks. Several people with different expertise work on these logs every day.
You won’t find many competing services where so many resources, both financial and human, are spent on analyzing security threats and improving protections.
Superior data backup
Over the years of operation, we’ve come to understand that while a good data backup is essential, it also determines how much faster they can back it up in the event of a problem. Regular backups are fine, but it’s the ability to restore the service quickly enough that a particularly demanding customer will appreciate.
We have invested a lot of money and time in finding the ideal solution. A lot of solutions that were developed over many months were simply discarded because the result was not the right one. Unfortunately, the actual operation of such a large number of services cannot be effectively simulated.
In the end, we ended up with the HPE 3PAR StoreServ Storage 8450. It is a repository that uses AI to optimize performance and predict problems. On paper, these storage devices can handle up to 3 million IOPS with a response time of under 1 ms. Hewlett Packard Enterprise guarantees data availability up to an incredible 99.9999%. They can scale up to 80 PB per system. In real operation, we managed to reach just under 1 million IOPs.
One of the goals was to get away from software and layers that hinder performance as much as possible. Just leave the hardware and firmware there, let it take care of everything. In total, we have 13x HPE 3PAR StoreServ Storage 8450s in the datacenter, 12 of which are actively deployed and 1 for development.
As a result, everything is faster and new possibilities for backing up and, more importantly, protecting your data open up.
Data protection
The topic of data protection is one that we have been discussing a lot in recent months. The current concept of where we would like to get to is as follows:
So what is it. The service data will run on the first 3PAR (in DC1/DC2). It will be used to make a live copy of the data to the second 3PAR (in DC2/DC1), where data will be stored 6 times a day (every 4h), 7 times a week (every day), 4 times a month (once a week) in the form of disk snapshots. These snapshots will be stored on “cold storage” in the form of a third 3PAR that is offline and located outside of our datacenters. At regular intervals it connects to the network and downloads data from the other 3PAR. Moreover, already stored data will not be deleted from the third 3PAR (old data will be periodically cleaned within internal processes, but what is not old cannot be deleted).
This concept is currently being tested. However, non-guaranteed data is already stored on the first and second 3PAR according to the infographic above.
What we are currently still working on is how to recognize encrypted data. That is, data that has been compromised by ransomware, directly at the 3PAR level. Once we do, we can detect a ransomware attack and not back up the encrypted data.
Oversized and redundant infrastructure
When you design and build a datacenter yourself, you know about every cable. At the same time, you’re always wondering “what if” and replaying various disaster scenarios in your head. The result is a robust, oversized infrastructure that is resilient to both realistic scenarios (fire, explosion, widespread flooding in the area, blackout …) to completely catastrophic ones (10,000 years of water (does that even exist?), total quarantine, lunar blackout, tornado, extreme temperatures (hot and cold), zombie invasion …)
There is a solution for everything. You just need to think it out and invest time and money into it.
Some examples:
In both of our private data centres we use flame retardant (fire retardant) cables for electricity distribution. In the data hall, a busbar system is used, which both simplifies installation and provides protection against fire. Both the cables and the busbar system are several times more expensive than the commonly used “flammable” solution.
The WEDOS DC2 data centre has been built from the ground up with flame retardant cables and busbar system, the WEDOS DC1 data centre has undergone extensive modernisation in recent years and the old solution has been replaced by flame retardant cables and busbar system.
In general, for both datacenters we have done our best not only to prevent a fire, but also to prevent its impact (the individual parts are separated by fire and the burn-through time is 30-120 minutes). If something goes wrong, we’ll deal with it quickly, air it out, replace what’s needed and move on.
Having two motor generators in a datacenter is pretty standard nowadays. Not that each of them can single-handedly run the entire datacenter, including the infrastructure. In WEDOS DC1 we have two specially built to measure (parts from different manufacturers, one smart and one dumb).
Each of them can completely power the entire datacenter for 12 hours of full power (16-17 hours at current power). Both are built for standard continuous operation (they can be refilled on the fly) and one is even built for true non-stop operation (they are used on ships, for example, and have to run for months at a time). We even considered with him that if the price of electricity continues to rise at this rate, we will just start burning cheaper diesel. For months at a time.
What about the 7 motor generators? Anyone capable of tightening the entire datacenter. 2 on the roof, 3 inside the building. WEDOS DC2 is built from the ground up to operate in extreme conditions.
It may seem like an unnecessary “luxury” to you, but the truth is that we want to be able to cope with something very bad happening, or to be able to start full operation as soon as possible (within a few hours at most).
With us, you also pay for the fact that if an extreme situation occurs, we will be able to get everything up and running very quickly.
We’re still looking for someone 🙂 We are currently looking for new colleagues for 8 positions. But it’s not because we are short of people to run it. If we just wanted to keep everything going, we only need about 1/3 of the people we have now. The rest just does the development and moves everything forward. We are looking for people to grow faster or to start other projects.
However, the fact that there are only 1/3 of us to operate has one huge advantage. When there’s a problem, like Covid, there’s always enough people to cover that 1/3. In addition, we have 2 separate buildings, with a total of 3 separate office spaces (2 offices + 1 meeting room). We were able to function that way in Covid. We divided our colleagues into groups so that the groups would not meet.
We managed Covid quite well, although we were worried that we would have to cut back on customer support. Plenty of people and your own space will come in handy. Everything has moved forward.
Experienced colleagues
If a new colleague stays with us for more than a year, he usually stays. Our team is gradually growing and people are more experienced. It’s noticeable, especially when a new problem is being addressed. There are more people trying to look for what’s going on and sharing information with each other. We used to rely on a few experienced colleagues. Today, as everything has grown, there are more ways to solve some of the new problems. The confidence to “try something”, which comes from practical experience, is also well known.
The above applies to new problems and some specific situations. Common operational things (broken hardware, DDoS attacks, etc. ) are handled by each department. He has proven procedures for this.
Given the volume of services we run, everyone gains experience quite quickly. Many tasks are repetitive, so there is also a strong push for automation, leading to faster and more efficient processing of requests.
Experienced people with years of experience are priceless.
Guarantees verified by a third party
As you know, we have a number of ISO certifications. ISO certificates are a guarantee that not only our services, infrastructure, but also our company processes meet international standards.
We currently have:
- ISO 9001 (quality management system)
- ISO 14001 (environmental management system)
- ISO 27001 (Information Security Management System)
- ISO 27017 (set of practices for information security measures for cloud services)
- ISO 27018 (set of practices for protecting personally identifiable information in public clouds)
These certifications are not cheap and require quite a lot of administrative work. We prepare for the whole event all year round, but the month before the ISO recertification and audit is of course more intensive. The last audit in March 2022 took place over 2 days.
For 9001 and 14001, there is a surveillance audit that has to be done every year to see if we are compliant. Recertification is underway for the others.
The certificates are a guarantee that not only our services, infrastructure, but also our company processes meet international standards.
Our most demanding customers often have to address specific hardware, infrastructure and security requirements with their clients. Most of the requirements are based on ISO standards. So when we have these certificates we don’t have to deal with individual contracts.
Currently, even the cheapest shared web hosting in our country is operated according to the above standards, which is sufficient even for demanding customers who run their clients’ applications here, which are often companies with billions of turnover. Plus, it’s often enough for the authorities 😉
Yes, you pay for that too and you don’t even know it 🙂
Conclusion
We strive to move our services forward. But it requires work on many levels. You can’t just quickly push what is visible. It is necessary to solve the background (datacenter, buildings), infrastructure, hardware, development and people who will take care of all this. We try not to rush anything.
We are currently working on a number of amazing projects that are unrivalled in the Czech Republic (for example, WEDOS OnLine and WEDOS Global). Gradually we will launch, promote and move forward. They are Czech projects. We believe that they will get not only our brand but also the Czech Republic into the world 🙂