Explanation of the problem with DNS records from 28. June 2014.
Explanation of the situation
Dear Customers,
I (I, Josef Grill, as the Chairman of the Board of WEDOS Internet, a.s.) would like to apologize for the failure of some interfaces for the management of our services. It was not a server outage, but a problem caused by a DNS change on one of our domains (wedos.net), which is not actively used (no website on it), but is responsible for redirecting FTP accounts and mail accounts and access to PHPMyAdmin for some services. This domain inadvertently expired and our registrar, with whom we have registered it, changed the DNS servers. Immediately (within minutes) the technicians fixed everything that night. The effects of the DNS change have been delayed by some ISPs, causing some clients to experience problems accessing the above services.
Apology
This is my personal failure, when I took over the renewal of company domains from a colleague (until then there were two of us to be sure) and in the “heat” of work I overlooked the need to renew the domain. The domain has not been kept in our automated system identically to customer domains, but is kept outside the normal operation of the company to prevent problems and misuse of the domain. I completely wrongly assumed that there was an automatic renewal for the domain that we have in our system for our customers. This is a personal failing of mine for which I am very, very sorry. Unfortunately, it’s like a bad movie and I should probably go very deep underground…
In total, about 250 clients contacted us today, and the change has taken much longer to take effect. There will be a few more clients affected, but I can offer nothing but apologies and compensation. For individual compensation, please contact us via the contact form. The bulk solution will be in the administration on Monday evening.
Overall, all servers were working and the problem affected a small part of clients, which can be seen from the data traffic from our network, but this does not excuse us (me) at all.
Ongoing changes in our processes
In order to avoid similar problems in the future, we have decided to connect all our domains to our automated system to avoid human error in the future. In addition, we will extend them right now for the maximum possible period of ten years, because we are only at the beginning and nothing will deter us from continuing. We intend to continue to bring you new services and improve existing ones.
We are certainly not going to hide behind any excuses and hide behind any mistakes of someone else. Due to technical complications in May, we are preparing to deploy a new monitoring system, where you will not only see any hesitation of our servers directly in the administration, but also automatically alert you to any problems with any of the services operated. We will keep you informed about other modifications that have taken place or will take place in the coming week, both on our website and by e-mail to all clients. It was already apparent today that we had changed some things, as we were communicating all the time (including on social media) and also had a message (with instructions for a quicker solution) right in the administration and in the contacts on the website and chat login page. As promised, on Monday evening we will have information in the client administration about compensation for the problem from May and we will add compensation for today’s situation. Mistakes are paid for and so must we.
Just in the past few days we have decided to become a registrar of transnational (generic) domains and we will manage both our and your domains ourselves.This will free us from third-party services and thus, for example, in case of domain expiration, we will not set up a foreign DNS server, but we will “hold” the domains for clients as much as possible. Not only will this mean better prices for you, but we will also be able to help you much more in both technical and legal matters.
Some information on the situation
The problem affected a few tenths of a percent of our clients. These are clients that use DNS servers from providers that cache DNS records for long periods of time. Unfortunately, this is (in this case) a very unpleasant feature of DNS records that we cannot control and so we have to “drink” the “cup of bitterness” to the bottom and everything should work without problems for everyone during tonight (24 hours should be enough to reload DNS records). That makes the situation all the more unpleasant and all the more regrettable.
It was not a technical problem or a safety problem, but my personal failure.
I apologize for the inconvenience. I am very uncomfortable with this, especially as we dealt with our (first ever) technical issue last month. As part of the remedy, we replaced the UPS, got another motor-generator, and modified other things, but I’ll really write about that next time.
I am constantly pushing my colleagues to have all servers in fully functional condition, which they are doing (we run well over 300 physical servers). We have had virtually no problems in the last month (apart from a problem with one of the mailservers in the last few days), just like we have for 3 years before that, and we are doing our best to keep everything stable. We’ve already solved the mailserver… And I’m gonna screw it up like this…
Once again, I am very sorry to all affected customers and colleagues and other shareholders for the complications caused by me.
Conclusion
I want to assure you that you are our priority and that is why we are completely honest with you. During the day, colleagues primarily advised clients with access problems on how to resolve the situation most quickly. That was a priority and now we have issued this statement.
We trust that you will retain your confidence in us despite this hesitation. In return, we promise to do everything we can to make our service work perfectly.
A plea in conclusion
In case you want to discuss under the article, I would like to ask you to state your full name and also the service you have with us. I have faced up to my (inexcusable) mistake and so I expect that an anonymous discussion with proxyserver contributors will not ensue. I believe that all the debaters will not be ashamed of their sites and their name.