When we launched VPS ON and the new NoLimit on the cloud solution and HPE Moonshot server enclosures designed for large clouds in the fall with much fanfare , we thought IPv6 deployment would be a matter of months. We had no idea, however, how many complications we would encounter.
Some of the history of IPv6 at WEDOS
Since we started much later than most of the big players in the hosting market, we only got a minimum of IPv4 addresses. Europe has run out of IPv4 addresses and has started to save a lot of money. We had to fight hard for each range and work hard to prove that we were making fair use of each of those allocated. Anyway, we had to buy more ranges for a lot of money. We can write about that next time.
This was one of the main reasons why we promoted IPv6 from the beginning and hoped for its rapid deployment. But as it turned out, IPv6 brings complications and the world was not yet technologically ready for them.
That IPv6 is not (yet) the future we have definitely given up 11. August 2016, when our network infrastructure was hit by a very strong IPv6 attack and caused partial unavailability of our services. At that time we already had our DDoS protection, although it can’t compare to today’s. If the attack had been conducted over IPv4 we would have handled it without any problems. But he wasn’t. On the other hand, we also dared to run some advanced experimental protections. In hindsight, this incident put us about six months behind in deploying advanced protections. For more information, see our article Statement on the connectivity problem with VPS on 12.08.2016.
We quickly had to dig up the network and split the traffic into IPv4, which we could filter, and IPv6, which we could simply cut off in the event of such a large attack. We also changed our pricing policy and started to offer IPv4 for 1 CZK. Previously we had them for 50 CZK and we used their sales to finance the purchase of new IPv4 addresses.
Since then we have been very careful with IPv6.
Network Infrastructure Upgrades 2017/18
If you follow our social networks, you know that we have embarked on a major upgrade of our DC1 “Bunker” for about 20 million CZK. It involves drilling the building like an easel and completely redesigning each part of the infrastructure to make it as independent as possible from one part of the infrastructure to the other.
It is not just about redundancy, which is common in better datacenters today, but to be able to fully function if one part of the infrastructure needs to be not only serviced, but also to perform demanding long-term technical modifications. Thus, the individual branches must be sufficiently oversized to cover the entire datacenter operation in the long term.
With the design and construction of DC2 “Podskalí” according to TIER IV requirements (the certification process has already begun), we have moved to a completely different level. We are thus applying the new insight and knowledge to DC 1. For more information, see the article WEDOS 8th Birthday or what’s new, what’s been successful and what’s not so successful.
The modernization of the network infrastructure was rather a complete reinvention of everything from the ground up and a transition to new hardware. At the end of 2017, we connected the first 100 Gbps route, and two more in 2018. Of course, the cloud solution we have built also has much greater demands. If hundreds of physical servers are to become one and everything has to work in real time, the requirements are somewhere else. And we’re not just talking about speed.
How did we (not) do
IPv4 didn’t give us any problems. Almost everything went according to plan. However, with IPv6 we have encountered one problem after another.
If you want to offer a full-fledged cloud, you have to be prepared for your customers to start creating hundreds of virtual desktops. Our cloud can do this automatically or via user scripts. While with IPv4 everything is handled through one central virtual machine, with IPv6 each virtual machine can have its own public IP address.
That means you need a router that has incredibly large routing tables. Nowadays, of course, they are already like that, but they cost as much as a better car or even as a house…
Fortunately, WEDOS is already a fairly well-known brand and so suppliers regularly offer us hardware to try before we buy. They even agreed to our request to try two of the same pieces at once.
This was an important requirement because we want the routers to be able to synchronize routing tables with each other.
As it turned out after some time this did not work as we wanted because of IPv6. We have addressed this with the manufacturers, who have promised a remedy. There was a lot of correspondence, firmware modifications. In the end, nothing helped and we lost 3 months.
So we started looking for other routers that would meet our requirements and not have a problem with IPv6. We finally found what we were looking for. A router with a switching capacity of 800 Gbps and/or 720 million packets per second and 350 thousand entries in routing tables, of which 128 thousand are networks /64. This is probably the best router (in our case L3 switch) from HPE in 1U size.
After successful testing, we placed the order and another curiosity happened. Somewhere during transport, the fans fell off the pallet. It took a few more weeks to get this sorted out and we were able to start more tests and then plug them into one route and then the other.
Of course, it didn’t end there. We had to redesign our internal system. Also for security reasons, the customer has always been given specific IPv6 addresses from their range to the machine. He must have picked those up as he went along. It didn’t bother me with VPS, but with Cloud, where you will automatically create dozens or hundreds of virtual machines, it’s annoying. That’s why we modified the system to work with ranges.
Another modification of our system was related to one of the latest innovations and that is proxy servers. A and AAAA records will have multiple proxies. If we want to offer services with high availability, we can’t do it without proxy servers that will distribute content from multiple machines.
And of course, it was necessary to make adjustments in our administration.
We could go on like this. We are preparing everything for operation in two datacentres and this brings additional modifications, which are more complex with IPv6 than with IPv4.
So where are we
We’re done! We will not be redoing any cables at this time. Both routes are ready and fully functional. This is the final solution.
The entire network infrastructure is more robust and resilient. We can even filter IPv6 attacks much better and more efficiently.
What about services?
IPv6 for NoLimit web hosts is ready and a non-public beta test is currently underway. Well, now that we’ve revealed it, it’s public 🙂 Just email my colleagues via the contact form that you want IPv6 to your NoLimit and they will set it up for you. Although these are public “tests”, the NoLimit with IPv6 has for example already our web wedos.cz and wedos.sk. Yes, our websites run on NoLimit web hosting 🙂
By the way, our new websites are running on WordPress, but more about that next time.
We don’t want to make any promises with VPS ON. Although we have resolved all the issues, we still tiptoe around this service and any adjustments are made with the utmost care. So the technicians don’t want to rush anything and we’ll leave it up to them. Basically, all that’s missing to make IPv6 operational is registration.
WEDOS disk runs on IPv6 without problems.
Dedicated servers, VPS and VPS SSDs are already IPv6 capable from the start. Specifically for VPS, we just improved the filtering options.
Conclusion
There was a lot we had to do to get to where I am now with IPv6. For both software and hardware, we ran into limits with IPv6. In retrospect, it seems that many hardware and software vendors are treating IPv6 as a toy for ordinary users rather than something that should replace IPv4 on a global scale.
We are the leader on the Czech market, we operate the most hosting in the Czech Republic and so the size brings that we encounter new problems, which are the limits of various routing and switching tables. We always find a solution, but sometimes it takes time. Well, I hope we’ve explained it nicely and humanely.