The power went out in Hluboká after 4 pm today. The length of the outage was just over an hour. While neighbors came out of their houses to make sure it wasn’t just them, most of us didn’t notice anything. The only thing that alerted me to the outage was an email in the mail and the noise of the motor generators.
In the first datacenter DC1 we have a total of 3 motor generators. However, everyone can tighten the entire datacenter independently. The others are for backup.
All of them are technically built for continuous operation. In contrast to the conventional design for a backup generator, we use those designed for ships or remote research stations where there is no power line, for example. By the way, in the second DC2 datacenter there will be 5 of these motor generators 🙂 There are already 4 installed.
Again, each of them can handle 100% of the load. So all traffic is backed up several times in each datacenter.
It is also interesting to note that our generators are made of different components and different sizes to avoid possible defective series, etc. Another interesting thing is that in the second datacenter there are 3 inside and 2 on the roof. And I’m sure you don’t know that the ones inside are designed so that each is cooled differently (one has its own air conditioning, one has cooling on the roof, and one is cooled by a liquid circuit).
Each of the engine-generators has a supply of diesel for 12 hours of future full operation. So each datacenter lasts tens of hours without refueling and only then do we have to refuel.
Motor generators are checked once a week (Monday morning), then tested once a week to see if they work. This is always on Thursdays at 10:00 for a minimum of 30 minutes. And the highlight is the regular stress test. This is once a month (first Thursday of the month at 10:00) and then the electricity is dropped for real and we observe whether everything works as it should and the test lasts 30 to 45 minutes.
Believe it or not, but there is a difference between testing generators just to see if they start and are off load and on load. That sharp test will reveal every little thing and flaw. Once we found problems on the UPS, another time in the power switching sequence. At other times we see that there is a drop in cooling. This live test is very rare and most datacenters do not do anything like this, much less regularly every month.
All generators have remote management and control. So we have an online overview of their status, operation and operational values.
Of course, the generators power not only the servers, but also the UPS batteries and cooling.
Maybe next time we’ll give more details or more power details. For example, it is interesting that each datacenter has 2 power branches and each is composed of components from a different manufacturer. Again, to avoid a manufacturing defect… 🙂 But more on that next time.
Do you know of a similar solution elsewhere? We also visited Temelín NPP and we have nothing to be ashamed of 🙂