So it’s been eight years since we officially launched WEDOS. Since then we have come a long way together and changed the Czech hosting market beyond recognition. Yet we still feel that we are just getting started. This is largely due to the new technologies that we are testing and gradually deploying, the ever-growing team of skilled people who are passionate about trying new things, and the construction of the second datacentre, which is still forcing us to think differently. Everything just smells so new 🙂
But let’s take it in order, what has changed in the last year.
Let’s start fresh here on the new website. Based on WordPress, by the way.
Datacentre WEDOS 1 “Bunker”
It all started in a former civil defence shelter located in a strategic location. In 2008 we started to convert it into our first datacentre. At the time we didn’t believe we could fill it ourselves, so we considered serverhousing.
Soon after the launch of our services, however, it became clear that the market would grow significantly. A new breed of customers has emerged who want to use open source content management systems. They are not afraid to learn and can install, manage and extend it themselves. The Czech market was not fully prepared for them. Shared web hosts had many limitations at that time, for example, low memory limit for running scripts and there was also a problem with data space. The competition often assumed that a customer with a project the size of a content management system would have enough money for an expensive service, because it must have cost a lot to program something like that. Our NoLimit, on the other hand, had literally “overshot” parameters for that time. In 2010 the value of memory limit 128 MB for shared hosting for 15 CZK (at that time there was a big recruitment event)? Database space 1 GB? There was more. Add to that the dissatisfied customers of the competitors, and the stage was set for some serious growth 🙂
We have improved our “Bunker” several times since then. The biggest intervention was cutting through over ten tons of reinforced concrete for the new Freecooling in 2012/2013. However, this is nothing against the current modernisation. In total, we decided to invest CZK 20 million and to move our “Bunker” forward. The modernization will significantly strengthen the entire datacenter infrastructure.
We are rebuilding the first datacentre and building the second one so that both will meet the demanding TIER IV criteria. The second is already going through the certification process. It is the result of thousands of hours of studying documents, manuals, expert work and, of course, planning in collaboration with experts. When you invest so much time in something, it moves you, or rather kicks you, somewhere else in the industry. Then you can’t look at a “regular” datacenter the same way. It will completely change the way you think and look at the solutions you use.
We have incorporated all our experience into the modernisation and it shows. When we converted the civil defence shelter into a then state-of-the-art server room in 2010, we thought what chaps we were 🙂
Today, our thinking is much further along. We see the technologies that need to be properly arranged around the servers to minimize any disasters, and across the entire building. You’re not just making a data hall with technologies, but a kind of data building riddled with technologies through and through according to a clearly defined logic. Plus, you’re not limited to the standard stuff. You are replacing them with modern solutions that are much better suited to the concept of the whole solution. Often these are technologies designed for other industries, you just get them custom made 🙂
But enough with the words. See for yourself…
Because of the progress, the building had to be drilled through 50 cm of holes through and through. You just can’t get by with a data hall 🙂
Nowadays, modern technologies such as the busbar system are available. It hides all the electricity, is more durable, fireproof, easy to make additional sockets if you need them, can be expanded after installation, and you can distribute much more current with it than through cables.
Building individual branches, without any crossing, seems like a small thing, but it’s a lot of work and additional cost. But when you do it a few times, you don’t even think about it. Just count everything in units of centimeters 🙂
It’s all about the relatively high cost of investment. Everything is also made with great fire resistance. We are now making the current DC1 datacenter to be fire resistant for 30 minutes and all elements are separated from each other and everything is completely dual (and without a single concurrency and without a single crossover). In the second data center, DC2, everything is built to fireproof the entire operation for 90 minutes. Everything is also minimally doubled… This is a truly unique solution. When you consider that some elements (for example, the electrical busbar system) are certified to withstand 120 minutes and some elements are completely flame retardant and self-extinguishing (for example, all electrical wiring). Of course, we have a fire extinguishing system (gas in DC1 and eventually a combination of aerosol and nitrogen atmosphere in DC2).
Some technologies you know will pay off, others you will test over time. Freecooling was one of the best investments we made. It uses heated air from the server room and mixes it with cool air from outside. Thanks to this very efficient and environmentally friendly cooling, we can keep the price of our services very low in the long term. Part of the modernisation is its substantial extension. Originally he was mixing an average of 30,000m3 per hour. It is now running at 50,000 m3 per hour.
It’s been a lot of work and we’re still going. As our trained electricians wait for more technology to come to DC2, they continue to build electrical branches in DC1. Yes, we do as many things as possible ourselves, because that’s the only way we can solve problems immediately. We know about every cable in our datacenters. After all, we will soon be operating two private datacentres that we have completely designed ourselves. For this you need your own trained people with years of experience. We employ her directly ourselves and are thus not dependent on external suppliers.
The modernisation of DC1 has several phases. We want to finish the first one by November 2018. One independent branch of electricity and connectivity will be completely upgraded. At the same time, we will replace older parts of the infrastructure such as motor generators, switchboards, cables, UPS and batteries with brand new and more powerful ones. The second branch will be upgraded in spring 2019.
DC2
The construction of our second private Datacenter WEDOS 2 alias “Podskalí” took a long time. We don’t have to rush anywhere 🙂 We have enough time for everything and we follow the path of perfection rather than rushing something.
This is a unique project where all the servers will be immersed in an oil bath from which a water circuit will transfer heat to a nearby municipal pool. In addition, we want to be the first in the Czech Republic to receive TIER IV certification for Operational Sustainability from the Uptime Institute, sometimes referred to as Gold TIER. If we succeed, our second datacenter will be only 13. datacenter with this certification in the world. And that’s saying something 😉
At the end of August and the beginning of September this year, we sent the first official documents to the Uptime Institute. Lots of documentation describing our unique solution. Let’s see how they like it.
The construction of the datacenter itself is almost done, just a few last things are missing 🙂
In fact, we are already at the stage where the individual technologies are starting to be connected and test operation is approaching. We would like to start this by the end of the year. We plan to launch the first test “oil” services, but without guarantees, at very reasonable prices 😉
And the launch? We’d rather not promise that. There are still a few tasks to be completed which depend on administrative matters and in particular on some subcontracts. On the other hand, we believe in ourselves, because this year we managed to “unobtrusively” dig up the castle gardens in the state castle Hluboká Nad Vltavou and connect it to 100 Gbps fiber optics 🙂
Of course, we had to do everything with the utmost precision and put everything back to its original state. The photos show the condition before everything “sat”.
We owe Open Source a lot and we would like to pay it back. We have previously supported various conferences and open source community gatherings. We know how difficult it is to organise and promote them. We were on site and talked to the organizers.
That’s why we have a state-of-the-art lecture room for 50 to 60 people in the second datacenter. Here the participants will sit comfortably, have the biggest screen that could be put in the room in front of them and listen to the lecture with a high quality sound system. In addition, everything will be streamed online and a record will be made for others. We plan to provide it completely free of charge for the open source community.
We would like to have an event here at least twice a month, including smaller open source projects. We don’t mind if 10 people come. With streaming and online recordings, it will be for everyone. The most amazing things are often used by only a few of us 🙂
Moreover, Hluboká nad Vltavou is an incredible place for rest and relaxation. On Saturday, we’ll give expert lectures, go on a tour of both datacenters, then head to the after-party and the next day you can go enjoy the tourist and sports attractions that attract over 750,000 people a year 😉
NoLimit
NoLimit is still the most popular shared web hosting service in our country. A couple of days ago we passed the 100 thousand active NoLimit web hosts mark. So no number of orders, but actual active web hosts that are running something. Moreover, the trend is still growing.
About 10% are already running on the new fast HPE Moonshot servers. NoLimit has been automatically set up on them since November 2017. So 3.4 Ghz processors and fast disks were very popular and many of you migrated because of the speed.
The service runs on a new cloud solution similar to VPS ON. Unfortunately, the new solution really bothered us and before we tuned everything to the current form, we were struggling with failures. But everything is already working fine. We’ve done it and we can keep innovating 🙂
WEDOS Antispam
WEDOS Antispam is the latest innovation that we have deployed now in September. It replaced an old solution that was no longer satisfactory and with which we were not very satisfied. The new WEDOS Antispam is about 14 times more effective in detecting spam. It is built on the best that can be used for detection today, it evaluates emails according to almost five hundred parameters, and it uses “self-learning” algorithms to make detection more accurate.
We invited you to two public beta tests to help us choose the right solution and debug it, and once again it proved how great it is to have a large community of fans who aren’t afraid to try new things and help us out. Thank you!
After deploying WEDOS Antispam, our customer support was literally flooded with questions about whether we had something broken. People used to deal with a few dozen spam messages a day and now suddenly nothing 🙂
It’s true that we have been busy deploying and developing other things, and we have somewhat forgotten about such an important service as email. We have learned from this and will focus more on emails in the future. In fact, there are already a few improvements planned:
- You can look forward to the introduction of the Bulk folder (provisional working title). All trade offers will be stored there. You’ll see them when you feel like it. Of course, you’ll have full control over the filters and you can set which bulk mail to not put in your Bulk folder.
- You too can get involved in machine learning. For now, you are sending us samples of problem spam manually. We have some automation ready, but it’s not quite what we want. In the next version you will be able to mark spam and solicited mail yourself, which we definitely should not put in spam. Your decision will then be received by our antispam and learned from. If some new spam appears that our WEDOS Antispam doesn’t already know, it will take a few hours for it to start marking it correctly. However, if more of our customers mark this spam as junk, that will be a strong signal. The entire spam campaign can then be over in a matter of minutes.
You can read more about the development of our antispam in the article New WEDOS Antispam.
Proxy server
For NoLimit, which is set up on the new HPE Moonshot servers, proxy server technology is used to cache static content. Nowadays, when most content management systems (WordPress, Drupal, Prestashop, etc.) can work very well with this technology, it is an ideal solution to lighten the load on servers and give you the opportunity to host sites with relatively high traffic. All cached content is stored on proxy servers. It does not therefore in any way benefit your service with us. If you debug your sites and teach them to use our proxy servers, tens of thousands of visitors shouldn’t be a problem. We can scale the proxy servers according to your needs.
A small example of how our proxy server relieved the hosting after turning on caching. In blue is the number of requests the hosting had to handle and in red is the proxy server. The test was conducted between 19:00 and 23:30. It was a highly visited large Czech eshop with a turnover of about 1.5 billion and operating on a classic web hosting.
This service is of course completely free for all our NoLimit customers 😉
DDoS protection
Building our DDoS protection has taken a lot of effort and millions of dollars of investment. But we had no choice. In the second half of 2014, we became the target of repeated and very strong DDoS attacks targeting our network infrastructure.
It wasn’t easy. In the beginning, we literally had to go to the store and buy several servers with the highest computing power and the most memory. It didn’t matter the brand, we just needed proper machines that could count right now :). Development and testing was basically 24 hours a day, all the time. When the first results appeared after deploying the automatic filtration, it was incredibly satisfying.
Every year, we move forward with DDoS protection by leaps and bounds. New sophisticated attacks are coming, and their complexity is somewhere else than in 2014. Sometimes it is literally fascinating to see how someone can design such a large-scale attack.
This year, we have been greatly helped by the deployment of 100 Gbps routes, through which we can have attacks passed from connectivity providers to our datacenter and filtered there.
Currently, unless it is something new, we filter all attacks to tens of Gbps and many millions of packets per second automatically and we only learn about their existence from the log. Stronger ones are rare. This year the biggest one had more than 20 Gbps (our probes stop counting at this number 🙂 because the processor can’t handle it anymore). This attack was on our website. The strongest one, which was directed to a specific customer, had over 15 Gbps.
But it’s not just about DDoS protection. We account for attacks in network design and network infrastructure deployment. We know that sooner or later someone will come up with something that will overcome our protection for a few minutes. By the time we can filter the attack, we’ll have a blackout. This is why we only ever lose part of the network, so you only experience a short outage before the packets find a new path.
IPS/IDS protection
Our IPS/IDS protection, which protects your site from attack, has also undergone several modifications. First of all, we modified some processes and involved more automation. We’re just more confident in filtering attacks. After all, we launched IPS/IDS protection in March 2016 .
There are currently about 20 thousand active filters – filtering rules. The number of botnets varies continuously according to botnet activity. We’re already actively filtering traffic from the outside. That is, if a customer service is attacked, we can limit it so that it does not do any further harm or, for example, cannot log into the botnet.
This is known when we have to suspend it for some reason (update, restart, service intervention, etc.). At that point, the load on the servers will increase significantly. Don’t forget that we’re not just protecting your website against attacks, but also against robots that are crawling every page on your website and extracting data for marketing purposes.
There’s an enormous amount of data from the protections. Tens of terabytes of logs. For now, they are only used to improve filtering, which is a bit of a shame.
We are considering making an additional service where you can access the data related to your web hosting. This service would include a number of other tools that we use. Not only would you have visibility into who is attacking you, but also which parts of the site are under attack, which would help you improve security and close any security holes.
We also have a service in place to restrict access to the site or VPS from only certain countries, or ban access from certain countries.
100 Gbps
For Christmas 2017 we gave ourselves the gift of wiring our first 100 Gbps route. It may not look so spectacular in the photo, but it was a really great experience for us at that moment.
Our competitors didn’t even dream about it and mocked us that it was impossible to have 100 Gbps connectivity in Hluboká. And surprisingly, we did. We have 3 different routes on Hluboká and each has 100 Gbps. Routes are via 2 different operators, going via 3 different routes to 2 different locations in Prague. From there we have connectivity through 3 different suppliers further out into the world. Currently we have 300 Gbps at Hluboká and 220 Gbps internet connectivity.
Of course, everything was preceded by a lot of preparation. It was necessary to negotiate terms with multiple providers. Many of us have many years of experience not only in hosting but also in building and operating datacenters. So when we design something, we never rely on the same technologies or series. So that’s why we have signed 3 contracts with 2 suppliers for 100 Gbps connection 🙂
For today’s technology, it is no longer a problem to get 100 Gbps connectivity to Hluboká Nad Vltavou, where we have our own datacentres. Of course, it was necessary to measure everything first to be sure. As it turned out, however, this was no problem.
You can’t stop the progress, and in a short time everyone who lives in a major city will be able to have 100 Gbps. The question is how to even test such connectivity, let alone use it 🙂
We’re clear. We want to start a real cloud with everything that goes with it. We have the right hardware to run large cloud solutions and the right technology. After all, our VPS ON service is a single virtual machine, which you will be able to set up manually, automatically or by scripting thousands. Unfortunately, we have had to deal with a lot of complications that these new technologies bring. So we are about 9 months behind with the development. VPS ON has been working for us for a month now without a glitch, so you can look forward to WEDOS Cloud, where there will be more options…
This will be followed by other upcoming services that we are preparing. You have a lot to look forward to. We want to expand not only in terms of operating in other locations (foreign markets), but we also want to bring new services. You can soon look forward to the aforementioned WEDOS Cloud, as well as dedicated web hosting for very demanding customers. We are also planning many new products offered as a service (SaaS)… but more on that next time.
Rebuilding network infrastructure
With the transition to 100 Gbps, we also had to gradually rebuild the entire network infrastructure. This process began with thorough planning. We considered multiple designs in light of previous experience (attacks, fault tolerance, ease of maintenance, scalability, redundancy).
All in all, including planning, it took us several months to rebuild the network infrastructure in stages.
The result is separate connectivity for IPv4 and IPv6. Rack with older servers (running NoLimit set up until November 2017, VPS and VPS SSD) and new dedicated servers WEDOS – HPE ProLiant DL320e is connected via 2×10 Gbps and the rack with HPE Moonshots needs 2×100 Gbps because the individual cabinets have a 4×40 Gbps connection (running on them NoLimit set up after November 2017, VPS ON and in the future also WEDOS Cloud).
Of course, there are some related changes to the connectivity provided for services. For dedicated servers, we increased the connection to 1 Gbps with 1:10 aggregation with a 100 Mbps guarantee. The transferred data is unlimited in our country (Czech Republic, Slovakia, abroad). In peak hours, you can theoretically use 1 Gbps and transfer up to hundreds of TB of data.
The newly operated servers are connected 2 x 10Gbps to switches that are connected 4 x 40 Gbps to aggregation switches where we already have 100 Gps connections out to the internet.
GDPR
There’s never enough bureaucracy so there can’t be more 🙂 The new regulation from the European Union left us cold at first and we wanted to avoid it through one loophole. However, it turned out that processing contracts are absolutely crucial for our customers. We have therefore decided to be 100% GDPR ready and meet all the requirements of both GDPR and our customers.
There were five of us working on GDPR. The commercial department prepared the texts and modified the terms and conditions. We took advantage of this and prepared them for the planned new WEDOS Cloud services. The programmers prepared the system for individual processing contracts, their revisions and lots of checkboxes for each input of personal data. Part of the preparation included a long phone call to the UOOU to clarify some things and get everything right.
In the end, everything dragged on a bit more and we launched the new terms and conditions just before the GDPR deadline. What followed was a flood of emails, enquiries via the contact form, chat and even social media. We had to tweak a few things to make the processing contracts meet some of the individual customer needs. We have over 280,000 registered users (and over 450,000 active services) and many of them have their own clients with individual contracts, outsourcing solutions, etc.
In the end, we managed to satisfy almost everyone. Until next year, we still have things to do, such as the translation of the processing contract into English.
We have published two articles on the GDPR at WEDOS and your FAQs on the GDPR. Although GDPR is not such a hot topic now, he recommends reading.
VPS/VPS SSD
VPS and VPS SSD have long been services that make us very happy. They are among the most stable and there are virtually no difficulties with them. There are currently 1650 VPS and 3618 VPS SSDs active. In total, we have only dealt with units of problems with them this year.
Moreover, since we’ve dumped Microsoft, they’re increasingly using them just for Windows with their own licenses. As your reactions have shown, this was a very good step. We have not lost customers, quite the opposite. But we don’t know the exact number. What you run on your VPS and VPS SSD is not our business 😉
Of course, we are still thinking about how to handle the service. There is still demand from your side, but we are not planning to buy new physical servers. Space for VPS SSD is so limited 😉
VPS ON
VPS ON gave us all a hard time. There were many obstacles we had to overcome to get to the stage where the service is running smoothly. We solved compatibility, drivers, firmware, had to completely rewrite and recompile some things. In the end, we had to abandon some solutions altogether, because the software developers had not yet taken into account such modern hardware as we use. But that would make a feature-length documentary. Well, we’ve sorted everything out and we can move on.
We have a working cloud service, with all the cloud conveniences. We can replicate, migrate, and backup data between physical servers without losing ping. Everything is tested and works. We haven’t seen a single warning on the monitoring since August. Only one hardware failure on the disk was addressed.
And what are our next plans with VPS ON? As we have completed the network redesign to make it robust, redundant and resistant to various forms of attack, we will soon deploy IPv6 for all.
Next up is individual DDoS protection. Services will follow in order for VPS ON to compete with VPS SSD. Probably the most interesting thing will be the ability to split a dedicated processor into smaller ones. VPS ON run on fast 3.4 Ghz processors. Why not make a few 1.7 GHz or even 1 Ghz out of them? Not everyone needs a lot of computing power to run a regular website, but they do need multiple threads at the same time.
Dedicated servers
Last year we had the opportunity to buy 254 HPE ProLiant DL320 branded servers with very fast Intel Xeon E3-1231v3 3.4 Ghz processors for a great price. We took advantage of this and now sell them as dedicated WEDOS HPE ProLiant DL320 servers. Originally we had dedicated servers more as an additional service, but it is quite interesting to offer it in such a volume. We currently have 101 of them rented and we rent out several more each month.
That’s why we automated everything. The service is set up without the need for assistance from our technician. You just order, pay and in a few minutes you get the access data to KVM/iLO. So you have a dedicated server right away and without waiting.
Of course, we are wondering if it would be worthwhile to expand our offer with another line of dedicated servers. However, it is more a question of price. We got branded HPE ProLiant DL320 for a fraction of the purchase price, while under warranty. Our own datacentre and automation further reduce our operating costs. That’s why they are literally unbeatable prices.
We will see, everything is possible, maybe we will eventually build another datacenter because of dedicated servers. We already have the land 🙂
WEDOS disc
Unfortunately, we have not paid much attention to the WEDOS disk in the last year. It’s still such a “dumb” external storage. The only thing we’ve added is GIT.
On the other hand, the customers like it. They also appreciate that it is a service of a purely Czech company in a private datacenter in the Czech Republic. There are many so-called “Clouds” for storing files, but they are abroad.
Of course, we are considering how to take the service further so that more of our customers can use the WEDOS drive.
What we are planning but prefer not to promise a date anymore 🙂
We’ve made a lot of promises. Some things we didn’t have time for yet, others didn’t make sense technologically or economically, so we had to stop their development. Add to that the problems with VPS ON, and a lot of news was delayed.
Now that we’ve mastered VPS ON, we can take a look at what our developers and engineers are currently forging.
New monitoring system with push notifications
When there is a service problem, it is a problem to alert thousands of customers as quickly as possible. Tens of thousands are practically impossible anymore. Emails have their limitations, and text messages even more so. Of the current technologies, push notifications work best for us.
Our developers have already prepared the system and are now working on its connection with our internal monitoring. We want to eliminate the human factor from the process. No approvals, texts or confirmations. When your specific service becomes unavailable you will receive a notification, once it is back up you will receive another one.
Notifications will also inform you of any changes that occur to your service. You wanted an overview of everything, so we’ll give it to you.
Custom CMS
You regularly write to us saying that you would like to have some kind of editorial system under our management. While we do offer an application installer, where WordPress is required, many customers find it difficult to maintain for the long term.
We are preparing this service. It was originally intended to replace MiniWeb, but in the end we decided that MiniWeb just works fine and is enough for many of you.
The new service will be for anyone who wants to have their own website but doesn’t know HTML. With drag & drop, they build their own website and fill it with text in a modern WYSIWYG editor.
WEDOS Cloud
The development of WEDOS Cloud was delayed by technical complications with VPS ON. We have solved everything and now we can focus on WEDOS Cloud. What can you look forward to?
- You buy the power and distribute it according to your needs among an unlimited number of VPS and they will be connected by their own private network.
- Charging for power consumed or on a flat rate basis.
- Possibility to create any number of individual VPS up to 8 threads of 3.4 Ghz CPU, 64 GB RAM and including a dedicated razor (the whole server will be yours).
- Additional service 10 Gbps connectivity (1 Gbps guaranteed).
- Automatic VPS creation by script.
- Creating VPS on different servers, rack, building.
- Possibility to connect to our DDoS protection.
- Other things that the sales department hasn’t been able to rewrite in a meaningful way because only hardcore IT cloud specialists understand it.
We are planning at least one Beta test where you will be able to try the service and help us debug it.
WMS
You’ve been calling for managed services for a long time, and we’ve been promising them for a long time. Due to technical progress, we have already basically discarded the two split variants. And we don’t regret it. What we can develop now cannot be compared to the previous one.
The whole service will be based on Docker, so you will be able to choose whatever parameters you see fit (some restrictions are planned). At the same time, take advantage of planned additional services, such as splitting a powerful processor into multiple slower virtual processors. For a regular website, this will help you to pull in more traffic.
WMS will have dedicated resources. You can also order the whole server (Intel Xeon E3-1231v3 3.4 Ghz (3.8 Ghz) – 4 cores, 64 GB RAM, 1 Gbps connectivity (aggregated 1:10)). Our entire NoLimit webhosting will run on it with the fact that you will have admin rights to set whatever you want (limits and PHP, MySQL functions). SSH is also being considered. Of course, it will be up to you whether you activate the advanced features. We want to give you the opportunity to break it all down 🙂
Overall statistics of WEDOS Internet a.s. current and since last birthday
We’ve prepared some more interesting statistics for you. For services, these are actually active services and not some fictitious numbers or numbers of orders, etc.
The data are as of 13. 9. 2018 morning.
| Number of employees | +- 35 |
| Number of cottages | approx. 480000 |
| Number of registered customers | 218.270 |
| Domains* | 298.184 |
| NoLimit* | 100.319 + 33.313 additional domains as alias |
| VPS and VPS SSD* | 1650 + 3618 |
| VPS ON* | 877 |
| Dedicated servers* | 152 |
| WEDOS discs* | 11.788 |
| * Active only | |
| Datacentres | 2 (one under construction) |
| Racky | 30 |
| Physical servers | about 1500 |
| Air Conditioning | 6 (including 2 backups) |
| Freecooling | 1 |
| Generators | currently 7 |
| Electricity consumed | 110-170 kW every hour |
| Number of completed orders | 205.213 per year |
| Payments booked | for example, 298,132 for 2017 and 254,663 for 9 months of 2018 |
| Flash drives sent | about 2 thousand |
| T-shirts sent | about 300 |