We have had another successful year of growth, development and implementation of many ideas that have moved not only us, but also the Czech Internet a step further. With a great team, we work on so many projects at once that sometimes it feels like we’re not moving at all, but you can just see the progress.
Datacentres
Since our last birthday, we have made significant progress in upgrading our first datacenter, DC1, and the second, DC2, is practically complete. We even had an Open House scheduled for the 13th in DC2. September, but unfortunately, it got bogged down in paperwork. Plus, it’s time to get ready for DC3 🙂
Datacentre WEDOS 1 “Bunker”
Our first datacentre has undergone many modifications, but none as major as the 2018/2019 upgrade. The first part of it is already over. The first branch with electricity was completely rebuilt. This has been significantly strengthened and the switch from conventional cables to a busbar system. This ensures maximum fire protection, greater robustness of the electrical network and easier handling thanks to the sockets that can be placed wherever we want. The first branch is located under the ceiling.
Our electricians will do the second electrical branch when they finish work on DC2. This will run under the floor and provide better redundancy. It was originally supposed to be done this year, but we can’t keep up.
We have also completed the upgrade of our freecooling, which mixes cold air from outside with warm air inside the server room. It has been serving us since 2013. It saves money on electricity bills every year because, unlike air conditioners, we don’t have to blow all the heated air outside. Another huge advantage is the ability to significantly increase the air pressure, so we can cool more demanding servers in one rack, saving space, or even servers that can not be cooled with conventional fans. It also extends the life of servers and their components, as it can maintain a constant temperature in the data room to within 0.1 degrees Celsius.
This year we have expanded freecooling significantly. So it takes up almost 1/3 of the data room.
This is what freecooling looked like in 2015
The vents have been extended to the racks and the entire technology will be enclosed in a separate room.
You can see the original freecooling on the right. Everything has been enlarged and power has been increased. Before the modernisation, it was mixing an average of 30,000 m3 per hour. It is now running at 50,000 m3 per hour.
The whole technology was so big that it had to be moved to the next room, which we used for research and development. Freecooling takes up a large part of it now. It also houses new switchboards, one air conditioner and a new UPS for the first route.
Datacentre WEDOS 2 “Safe”
We were already planning to open the second datacenter in September. Unfortunately, we are waiting for the last few formalities to be completed.
From the beginning, DC2 has been built exclusively as a private datacenter for our own technology servers only. Everything is adapted to oil cooling, i.e. the servers and other technologies will be immersed in an oil bath, from which the heat will be removed by a water circuit and through a series of heat exchangers to the city pool. We have already tested this technology and it’s just a matter of deploying it appropriately. But on this scale, it is a unique solution that you won’t find anywhere else…
At the same time, the Uptime Institute certification process is underway. We are targeting the highest “gold” TIER IV Operational Sustainability rating from the Uptime Institute. With the passage of time and after dozens of modifications in the documentation, we understand why nobody else in the Czech Republic has it 🙂 But it’s worth it to us, just for the feeling that we have perfected everything to the highest possible perfection 😉
As for the work inside, everything is already done. We have running water, the toilets work and of course we have 100 Gbps internet 🙂 The new datacenter has 3 different connections to the internet through our own fiber optic routes. Each route comes from a different location and enters the building from a different side. We just take care of everything.
Of course, there is still a lot of work to be done in the data hall, especially with pulling and connecting the electrical cables. By the way, electric cables are non-flammable, flame retardant and are about 5 times more expensive than conventional cables. The entire datacentre is built to operate for 90 minutes in the event of a fire in any part.
We also deal with such details that we have flame retardant lights without plastic. They are stainless steel, have a glass cover… We don’t look at design, we look at functionality.
The oil bath v4.3, which we introduced to you in March 2019, has already been placed on one of the racks. The stress tests in April 2019 were successful.
At the beginning of August we brought the first two test Moonshots to DC2. We placed them in an oil bath to see how they would be handled, how much room there was for hands and where to put the outputs in the end. Based on these findings, the v4.4 oil pan will be produced.
In the photos below you can see the whole moving process and the first testing.
We have almost finished the conference hall, where we will regularly organize various events, trainings and conferences. The basics are a large screen with a quality projector, a good sound system, a soundproof ceiling and a multimedia centre that will give the speaker all the necessary tools to make his lecture an unforgettable experience.
Everything is also adapted for recording and online streaming. There will be basically 3 camera shots. One will occupy the presenter with the screen, the second will be closer and take up more space from the side, for example for panel discussions, and the third is the presentation itself. This allows you to switch between cameras as you like when streaming online.
Datacenter WEDOS 3 “????”
We have known for some time that we are going to start working on a third data centre and the medium-term plan includes it. We have already purchased an ideal plot of land with a size of over 1 hectare. DC1 is slowly filling up. It can fit less than 70 racks, but we plan to place 52 racks there in order to preserve space for possible new technologies.
The DC2 will hold 10,800 physical servers, but they will all be in HPE Moonshot server cabinets and nicely loaded with oil. Which is very limiting. For example, if we could get a larger number of regular servers at a fraction of the price, like we did with the 254 ProLiant DL320e Gen8 v2. We’re now renting them out at unbeatable prices. We believe that for such great prices we could rent 5x as many dedicated servers. We just don’t have anywhere to put them.
We are also preparing a new storage facility for our WEDOS Cloud, where you will be able to buy space cheaply. We are currently testing a dedicated HPE Cloudline CL5800 Gen9 server with 512 GB RAM and 100 drives of 10 TB each, i.e. 1 PB of cheap storage space for you 🙂 Well, we can’t drown that one. Rotating discs don’t like oil 🙂
If we want to offer a cloud with huge storage, high availability and individual SLAs, we need to have at least two sites with servers like the HPE Cloudline CL5800. That’s why we need DC3.
However, DC3 is not planned as DC2 for TIER IV certification. We need a normal datacenter where everything is redundant and backed up. However, we will not, for example, deal with fire doors with 90-minute resistance, which are very difficult to find and make the whole construction very expensive. Instead, just get a standard fire door with 30-minute resistance. For TIER IV, there are plenty of things like this that you need to have custom made and often only a few companies in the world can do. Delivery times are long and prices are all over the place. So we will build DC3 on the basis of our experience with TIER IV. We’ll apply them where we can, but let’s not overdo it this time 🙂
In DC3 we are also considering offering serverhousing mainly for South Bohemian companies, but we already have interested parties from Prague. Some companies want to have part of the infrastructure outside the capital.
As for the concrete steps we have already taken for the construction of DC3, apart from the land, the project documentation is being prepared. We will apply for a building permit as soon as it is completed.
Technology
Technological progress in IT is relentless, whether it is hardware or software. If you don’t give a technology a chance, you will find that you are on the back foot and it may not be easy to catch up with the competition. That’s why we give everything interesting a chance. Especially some of the “undercounters” can be very fascinating 😉
Connectivity
When we wondered how much it would cost to run fiber optic cables between DC1 and DC2, we were not surprised by the price, but that it was a problem to find someone. They were all too busy and to be honest with the prize it wasn’t great either. So we decided to borrow the equipment and just blow the cables ourselves.
We had already dug different routes, through different locations, and put in a few protectors and microtubes so we wouldn’t be limited in the future.
Currently DC2 uses the connectivity from DC1. However, once they connect to our suppliers, which will be during September, DC1 will be able to use the connectivity from DC2. We just made another backup of our backups.
In total, we have the following lines from suppliers:
- 100 Gbps from CETIN via Písek
- 100 Gbps from CETIN via Tabor
- 100 Gbps from CTD via Havlíčkův Brod
Everything ends in different places in Prague and we have internet connections there:
- 100 Gbps from Cogent
- 100 Gbps from Telia
- 100 Gbs from Kaora
- 100 Gbps by Kaora in another location
- 10 Gbps from Telia
- 10 Gbps from CTD
In this respect we are the best connected private datacenter and definitely the fastest connected hosting.
We also ensure the robustness of the internal network. Most of the NoLimit services (set up since autumn 2017) and all VPS ON, WEDOS Cloud and WMS services are set up on physical servers with 2x 10 Gbps connectivity.
Our backbone network is fully redundant and our DDoS protections are also redundant. The backbone is built on 100 Gbps elements and then the individual network segments are built on multiples of 40 Gbps.
DDoS, IPS/IDS and WEDOS honeypot protection
As you know, our infrastructure and your projects are protected by our own robust DDoS protection, in which we have invested millions of crowns, and its maintenance, development, improvement and expansion costs us millions of crowns a year. But it’s worth it. Combined with multiple 100 Gbps connectivity, it lasts a lot. Since last birthday, the attack that we filtered was 37 Gbps and lasted 4 minutes and was directed at the VPS of one of our customers.
We’ve had stronger attacks, but this one was a record so far in terms of filtration. We have successfully filtered the infected traffic. In such strong attacks, it is common to do the so-called. blackholing and the compromised IP address is “deleted” from the world. At WEDOS we can filter it 😉
This year, we’ ve only been plagued by one new kind of DDoS attack. I mean, tormented is probably not the right word. That would be more of a story for the competition to tell 🙂 Once we found out what it was, we adjusted our DDoS protection. We were intrigued by this new type of attack and in investigating it we discovered a more efficient filtering technology.
This more efficient technology is incredibly simple yet effective. It can filter huge network traffic with minimal server requirements. We thought of building a new protection on top of it, which will draw data from our CML.
CML or (central log monitoring) is a giant database where all servers send their logs. For example, there are all access logs from every site on NoLimit. This is an incredibly huge amount of data, but we can process, evaluate and analyse it in real time.
In CML, we can then find attackers, according to a certain pattern of behavior, even very cautious ones, who can slip through our IPS/IDS protection. For example, you’d be surprised how many robots guess WordPress passwords using the 1 attempt per 1 site per 10 minutes method. However, it spreads its activity across multiple sites or IP addresses. Such an attack, even if it lasts for weeks, is difficult to identify. But if you have logs from 104 thousand web hosts, you will find that the robot sends hundreds of queries per hour. You wouldn’t see that on just the log from 1 physical server.
At first, we tested it manually in small volumes and uncovered dozens of sneaky attackers or bots carefully collecting marketing data. We then switched to automation, where the scripts look for specific behaviors and execute the blacklist. In total, we have found that we can block over 45% of the traffic that absolutely no one misses, even after a public challenge where many of you joined in the testing.
As for our IPS/IDS protection, which uses a number of methods to eliminate attacks on websites, we are working on filtering encrypted traffic. This is mainly a problem with content management systems, where attackers exploit known security holes. Our IPS/IDS protection automatically blocks such attacks, but not if you have an HTTPS site. We have a solution to this problem and will deploy it in the coming months.
You won’t find similarly robust and high-quality protection anywhere else. You can see the functionality even on a regular website that is not overloaded with robots. Today we block over 50% of all requests to our servers. And gradually the protection learns to block more and more types of attacks.
Proxy servers
Our proxy servers have significantly affected the operation of NoLimit. Who knows how to use them for caching, so he can have a website with hundreds of thousands of visitors per day on hosting for 33 CZK. And yes, we do have such customers.
And add one record holder who managed 180,000 accesses in 10 minutes.
For the time being, we do not limit accesses via proxy servers. We’re treating this as a test run.
WEDOS Anycast DNS, Anycast AntiDDoS, Anycast CDN
In the spring, we began seriously discussing the possibility of launching our own comprehensive anycast solution. We already have the complete technology and have tested it. We would be able to provide both content caching across all sites (just like our Proxy can do) and protection (DDoS, IPS/IDS). Including Anycast DNS, of course. We have calculated that it would be enough to put 45 physical servers in one location.
In the first phase we plan to select 4 locations in the world where we will place the servers. The fifth location will be at our place in Hluboká. In the next phase, 5 to 10 sites would be added depending on current demand. And in the future, there may be dozens or hundreds of sites around the world.
At the moment we have already solved everything for setting up Anycast DNS at RIPE. We got a new Autonomous System (AS) with new IPv4 and IPv6 ranges. As soon as we get our hands free with WEDOS Cloud and WMS, we’ll start building the prototype and start testing publicly.
Services
Domains
In February we became the largest .cz domain registrar and we continue to grow.
Unfortunately, we sometimes encounter problems with multinational domains that we are unable to resolve with our supplier, which affects our customers. In addition, he has told us that he intends to increase the price. That is why we are currently preparing our own ICANN accreditation. We have already taken the first steps in this respect.
Otherwise. besides the fact that we are growing and we are still registering and renewing domains at purchase prices, we don’t have much news 🙂
NoLimit
However, there have been many changes to NoLimit. After a long time, we decided to increase the important parameters so that it can again be among the best shared web hosts on the market. Even though it is the most popular shared webhositng on the Czech market for a long time, it needed to be improved after 9 years 😉
memory_limit: 128 MB → 256 MB max_execution_time: 90 sekund → 300 sekund PHP upload_max_filesize: 32 MB → 128 MB PHP post_max_size: 32 MB → 128 MB Počet aliasů: 1 → 3 Počet PHP procesů: 5 → 7
For the extra variant
memory_limit: 256 MB → 512 MB max_execution_time: 90 sekund → 300 sekund PHP upload_max_filesize: 128 MB → 256 MB PHP post_max_size: 128 MB → 256 MB Počet aliasů: 1 → 3
Of course, these are just the parameters of the service. In fact, we are continuously improving NoLimit.
For example, we were one of the first to introduce SSD not only for databases but also for content. Even today it is not entirely common.
As of the end of 2017, all NoLimit on physical servers with 3.4 Ghz (now 3.7 GHz) processors and NVMe SSDs are being set up. The elders will be moving in this year.
We have gradually launched for free for NoLimit for example DDoS protection, IPS/IDS protection or Proxy servers. And you can read what Proxy servers can do above.
This year, we’ve also modified NoLimit so that you can order it for a subdomain.
VPS
Customers on legacy VPS and VPS SSDs have dropped off a bit since we stopped doing events and stopped renting Microsoft licenses. Customers are either switching to our new VPS ON, where there are faster processors and disks, or they choose very cheap dedicated servers for 1000 CZK/month, which is a common price for an average or better VPS.
This year VPS ON finally made us happy 🙂 All the technical problems have been solved and the service is just going well.
Dedicated servers
We didn’t make any major modifications to the dedicated servers. We are currently considering whether there would be interest in some other configurations, but we probably won’t go into that now.
Otherwise, we started testing VIP/B2B services for dedicated servers. We already have our first customers.
WEDOS Disk
A long neglected service, but one that makes us happy. It is simple and works well. Customers like it mainly because of its simplicity.
Currently, it is most commonly used for backing up WordPress sites via plugins.
WMS
The WMS is basically already here. Technically, it is ready for launch and the last thing we are dealing with is business stuff. As soon as the management gives permission, the service will be launched in a public beta test. We don’t expect beta testers to encounter any problems, so we’ll open the service to others soon.
We are also considering a free trial period of several months. Especially if anyone would like to compare with the competition 😉
WEDOS Cloud
WEDOS Cloud is already in public beta test, but it has a longer way to go than WMS. It is a demanding and complex service, but one with very great potential. However, only true experts will appreciate it.
WEDOS OnLine
WEDOS OnLine is a service that we originally developed for our own use. It monitors all NoLimit at the level of individual sites (availability, response), for VPS and dedicated servers it tests the response via PING, but it can also monitor other parameters. The individual services are then grouped by physical servers, PHP version, updates, etc.
If, for example, a few % of sites on a physical server experience a significant increase in response, the technicians will be informed and can act immediately. We can detect congestion very quickly, even before customers start to experience errors.
Of course, in the case of detecting various anomalies that can be caused by a PHP version or a faulty HW series, this service is invaluable.
The public beta version can only measure web availability and PING, DNS, email (SMTP, POP3, IMAP) and FTP. With WEDOS OnLine you will be sure that everything is working as it should 😉 In the next few days we will have more news and then we will introduce the service to the public.
WEDOS Internet, a.s.
And what’s new with us? We’re still growing, tackling new challenges and working on lots of fascinating new things.
Unfortunately, not all challenges are pleasant. This year we have invested a lot of time and effort in the fight for a free internet. We pointed out the problem with Section 7b of Law 141/1961, which gave police officers the right to shut down the web for up to 90 days. None of our competitors joined us, which was very disappointing. After a series of articles and support from you, our customers and fans, we were heard enough that even the press started to take an interest.
- You want to destroy the competition? Czech laws are here for you!
- Let’s call it “lex WEDOS”
- We talked to Andrei about it. The situation is close to that in Russia….
- To help and protect? We’re gonna shut down your website. Brand.
We were even invited by MPs from the Pirate Party to explain the situation and tell them about the potential risks.
In the end it turned out that the police really can’t use the new law and one of our customers defacto lost his website. We weren’t afraid and wrote about it.
- The first website was deleted from the Internet on the basis of the new Section 7b of the Criminal Procedure Code.
- The first website shut down under §7b probably didn’t have to be shut down after all
And what did we win in the end? We received a promise by email from the police headquarters that they had instructed all police officers not to use the law and to send all disputed cases to the Special Activities Unit of the Police Presidium of the Czech Police, where there are trained personnel who deal with computer crime on a daily basis. Part of the promise was that it would arrive via data mailbox. But he hasn’t arrived yet…
Support
This year we embarked on a major project to upgrade our online support. Around 98% of customer support queries are requests that the user can click on themselves (send me a request for payment, how much does the service cost, how long does it take to set up the service, where can I find … etc.) or for the status of the service. These questions can be answered through a smart search that we are developing. He will search across our knowledge base, administration, tutorials and our new help.
Of course, it won’t work unless we create a robust database of every conceivable problem and how to solve it 🙂 For now, we’ve drawn mainly from the questions answered by our customer support, but that’s not enough. We also need solutions to problems beyond customer support and that is why the help.wedos.cz project was created. There are already over 2000 users on it who help each other, including us. But we have much bigger plans for the help.
Customer support is “tied” to a large number of skilled colleagues who would be much more useful in development as technicians or programmers. They just need to be engaged in learning and practical activities instead of chatting.
Our goal is to keep support to the bare minimum and offer our customers guidance and tools for everything they need.
VIP/B2B
This year we started testing VIP/B2B services. As the number of more expensive services in our country increases, so does the interest in an individual approach. For now, most VIP services revolve around dedicated servers, but we expect that in the future Managed WEDOS Cloud (we will set it up as you need it), Private WEDOS Cloud (all physical servers will be reserved for you) and individual WMS solutions (deployment, infrastructure design) will prevail.
We are still testing the whole program and the results are definitely not bad. Customers especially appreciate the individual approach and options we can provide.
Marketing
In January we introduced our energy drink WEDOS Energy. Its punk design has quickly become popular until we can’t keep sending them. We supply them not only for our services, but also for various conferences, events and sports matches. We’re slowly emptying our warehouses. Luckily another truck is on the way 🙂
We’re a little behind on the other gift items. The only new item in our repertoire is the WEDOS black women’s T-shirt with sheer.
We’ve slacked off a bit with the aggressive marketing that got us where we are. We’re gonna have to sit down and rethink this. We are preparing two super new services that definitely deserve a proper promotion that will not take the competition for granted. After all, the customer always makes the most money. Hmm what would you say if we give away 1000 of our WMS with dedicated HW if you give the competition, who only have shared (resold) services, the trash? 😉
Conclusion
Thanks to you, we have had another successful year. We’ll try not to disappoint you again in the next one 🙂
A few more statistics at the end:
| Number of employees | 38 |
| Number of cottages | 68.499 (-6.819) |
| Number of registered customers | 265.922 (+47.652) |
| Domains* | 318.116 (+19.932) |
| NoLimit* | 104010 (+3691) + 33,011 additional domains as alias |
| VPS and VPS SSD* | 4.653 (-615) |
| VPS ON* | 1.084 (+207) |
| Dedicated servers* | 222 (+70) |
| WEDOS discs* | 11.695 (-93) |
| * Active only | |
| Datacentres | 2 (one under construction) |
| Racky | 30 |
| Physical servers | 1500 |
| Air Conditioning | 2 x 2 ( plus 2 backups) |
| Freecooling | 1 |
| Generators | currently 7 |
| Electricity consumed | 130-190 kW every hour |
| Number of completed orders | 211,830 (+6,617) for the year |
| Sent by WEDOS Energy | 3960, another truck will arrive on Tuesday |
| T-shirts sent | 1780 |