We have previously introduced TIER in general and how important safety is for TIER. Today we’re going to introduce everything about power.
In the first part of the series, we discussed what is the concept of TIER in general.
In the last installment of this series, we covered how TIER assesses security.Today we’ll discuss power and everything around power. In the last installment of this series, we discussed what the term TIER is in general.
Power supply and connections
One of the myths about TIER is that a datacenter must have 2 independent power supplies – meaning 2 connections. It all comes from ignorance. How is it then? Each datacenter that is to meet TIER IV must have 2 completely separate power feeds to the servers. However, there can be one power supply, but it must be sufficiently backed up. How? Motor generators.
There are the same myths about substations. A data centre does not need to have its own substation or even several substations. It may or may not. If you have a power input of up to about 630 A, you can use a transformer station from the electricity distributor and take electricity at low voltage, but you can build your own transformer station and use the high voltage connection. That’s more of a business decision. For larger consumption you are usually forced to get your own substation.
The bottom line is that if you have a substation, you have to ensure its safety and if it is somehow accessible at first glance and you open it from the street with an electrician’s “wrench”, you better believe this is not the right security. When you have multiple substations, consider that you cannot “stack” them side by sidebecause if there is a problem on one – for example, a fire – you can be sure that it will affect the neighbouring one. You can’t say it’s in a concrete shell that doesn’t burn. Those who have seen a transformer station on fire will confirm that everything around it will be on fire and, moreover, you will automatically hit the other one when fighting the fire. Similarly, for subsequent repairs, you will have to intervene in the other…
The worst possible combination is if you put transformers next to each other and put motor generators next to them, which are also next to each other. Such a datacenter can never meet TIER IV. It only takes one fire of anything in this system and the entire datacenter is immediately without power. Even if it doesn’t burn down on its own (and believe me, it probably will), the fire department will “sprinkle” it on you. That’s why you need to keep everything separate and far enough apart. Ideally, you want to have everything fire separated as separate fire compartments with sufficient fire resistance, and also so that the power supplies are each on a different side of the building (and enter the building via a different route).
How are we going to have it?
The existing datacenter will have 2 power supplies. One source will be from the e.ON low voltage network and the other from the high voltage network, which will pass through our own transformer station (which will be on a loop of high voltage cables). Each power branch must be at 100% of the datacenter’s capacity and will also be backed up by a motor-generator (one per branch), with each motor-generator capable of providing 100% of the datacenter’s operation. For TIER IV, only 70% of the capacity is allowed to be used from the values specified by the manufacturer – Prime (maximum output during unlimited time but with variable load). So each generator still has tens of percent reserve.
The new datacenter will be connected to the e.ON substation, which is on the neighboring property, about 1 meter from our fence. The substation is connected to a high voltage loop of the e.ON network, which leads to various locations. A hydroelectric power station is also connected to this substation – to the same terminal block, which is able to provide power to 100% of the capacity of our datacenter. Inside the new datacentre there will be 3 motor-generators, each capable of providing 100% power to the datacentre.
Everything is different
After careful consideration and evaluation, we decided to go with completely separate motor-generators that would not be paired or synchronized in any way to avoid any potential technical problems. Everything will work fully automatically via the ATS (automatic transfer switch), which is a “simple” power switch that can only have two positions. It can either supply power from a primary power source OR from a backup power source. Never anything more. If the primary doesn’t work, it switches to the backup. If the primary comes back up, it switches back to primary.
We do not use and do not plan to use any smart boxes and contactors connected to them. According to various studies, contactors are 15-20 times more faulty than the ATS we use, which have a life expectancy (or average time between failures) of over 100 years. Similarly, our designed ATS have significantly higher numbers of switching cycles without failure than contactors. Our ATS have a minimum expected number of cycles without failure of 6,000 switching operations. So imagine that we should not expect a failure for 100 years, or at least 6,000 times a power failure (and subsequent switching of the ATS to a backup power supply) could occur before a failure could occur. For contactors, the numbers are significantly lower.
So what is the result?
Up to 3 power supplies could fail (or be serviced) at the same time in each of the datacenters and the datacenter would have 100% power to operate at all times! So the power supply is not just fused or double-fused, but even triple-fused. This is one of the most critical issues in IT operations and so this is very important.
Hydroelectric power plant
We would also like to mention that our datacentre will not only be extremely environmentally friendly, as we will be using very efficient oil cooling and at the same time the waste heat will be used by the local swimming pool, but our datacentre will be primarily powered by hydroelectric power.
The servers will be tens of percent more efficient thanks to oil cooling (they do not need to have fans that consume about 30% of the energy). At the same time, we will not need any large fans for cooling or air conditioning. Just economical operation using waste heat. If someone threatens that we have oil in there, our monolithic concrete building is designed as a catch basin and if we were to spill all the oil that will be in the building, we would have less than 15 centimeters in the catch basin and yet the basin is over 50 centimeters. But we’ll write more on this subject in more detail next time..
Do you know of a greener and more environmentally friendly project?
Power supply inside the building
A separate chapter is the power distribution inside the building. What is essential? The power distribution must be arranged so that all elements must be powered from 2 power branches simultaneously. So each server must have power from two branches (branch A and branch B in our diagrams) simultaneously, and each branch must be able to provide 100% power at all times and under all circumstances.
It is extremely important that all parts and all elements are separated and that branch A must not meet or cross branch B anywhere. Everything must be separated not only by some distance, but above all physically, so that a crash of one branch does not endanger the other branch. Likewise, everything must be separated fire-proof.
So imagine that you have to get 2 power supplies into each rack, but those 2 cables must not meet anywhere, must not meet in the same space, must not coincide in the same space, and must not cross. This is a fun one to try to draw and think about how you want to do it. This is where most datacenters end up because they are not physically built to implement this. Basically, no such datacenter is ready for this in the Czech Republic and so no one can get TIER IV. This may be one of the reasons why there is no TIER IV datacenter in the Czech Republic, Slovakia, Poland, Hungary, Austria or Germany…
It is completely unacceptable to stretch both power lines in one space at the same time – for example, under a double floor. This is totally unacceptable from a TIER IV point of view. As well as some crossing of routes, etc.
How do we do it?
One power branch will be located under a double floor that is non-combustible and constructed to meet the appropriate fire resistance.
The second supply branch will be located under the ceiling, covered with a plasterboard ceiling with appropriate fire resistance.
Before the two branches reach the floor, or the ceiling, they do not cross anywhere before, everything is separated, isolated both physically and fireproof.
The two power supplies will meet in the rack, but they won’t cross there either. One will come through a hole in the floor and be on one side of the rack (or cooling tub in the new datacenter). The second branch will come through a hole in the ceiling and will come into the rack from the top and will be on the other side of the rack (or cooling tub in the new datacenter).
The same rules apply to other datacenter issues – cooling (and cooling power) and connectivity inside the datacenter (we wrote about connectivity outside the datacenter last time). You must have no crossing, no overlapping, nothing in the same space anywhere. Our approach there will be similar to the power supply.
What else is essential with the power supply?
All parts of the power supply must be fully serviceable and always so that 100% of the power supply is still working and you must ensure that both power supply branches are working at the same time! It’s not so simple when you think about the fact that you have to be able to service motor generators, UPSs, switchgear, all the cabling and all the aforementioned switches (ATS). It’s not just that. It also took us a lot of work to figure it out.
In terms of servicing, it is not just about some regular checks and revisions in terms of the law, but it is also about preventing errors and monitoring the whole system. We could probably write novels about control and testing. Believe it, we check the condition of the wiring every week at the beginning of the week and every Thursday we test if everything is working properly and operationally. Every Thursday, we test whether a spare power supply will kick in. And on the first Thursday of the month, we always test the black-out. Yes, we always turn off the main power supply and test all systems for complete operation. After about 30 minutes, turn on the power supply and observe whether the opposite happens – all systems switch to the mains. This is also one way to prevent human error.
The entire power system must be capable of fully automatic operation. Everything must work by itself and react to errors and malfunctions so that no one, anywhere, has to do anything.
Motorgenerators
For motor-generators, the international standard ISO Standard 8528-1 is used. Each generator is thus rated in 3 values – Emergency Standby (the value of the maximum power it can handle in the short term – a maximum of 200 hours per year), Prime (maximum power during unlimited time, but with variable load) and Continuous (maximum power without time limit with continuous load).
Which generator can be used and with what value? It depends on the manufacturer. Continuous without limitation. Prime – only 70% of the capacity stated by the manufacturer. So you have to use a significantly more powerful motor-generator than you need for operation. It is assumed that there will be sufficient reserve for any problem.
Motor generators for TIER III or TIER IV must meet several criteria. Don’t expect any nonsensical values for how long the motor-generators have to run. It’s quite simple. Motor generators must:
- be designed for continuous operation, believe it is not quite common,
- They must have 12 hours of continuous operation for the entire capacity of the datacenter,
- again, they must be physically separated enough to ensure physical security (generators must be sufficiently protected and a fence is not enough)
- fire safety and it is therefore unacceptable to have engine generators next to each other or several metres apart without any fire separation and partitions with sufficient fire resistance.
This is a very common error and you see it practically everywhere, in all datacenters.
To be safe we decided to have each motor-generator from a different manufacturer, It is a complication with servicing, but it is safer, because we will not encounter some hidden serial defect in the future.
UPS
You must also have a UPS twice. Once on power line A and once on power line B. Each power branch must be able to provide 100% power to the entire datacenter. At the same time, you must be able to service or replace the UPS at any time, while ensuring 100% power on both supply branches at the same time. In this case, you will come to the point that you cannot, for example, use modular UPSs that can operate in N+1 mode. It is not possible to make a replacement there without removing the entire power supply branch. You must have N+1 UPS on each power branch physically to be able to service or replace anything.
It is taken for granted that the UPSs for the individual branches are completely physically separated in separate rooms, while the batteries are in separate rooms. The wiring for the different power supply branches must not cross, must not overlap and must not be in the same space.
It must be remembered that for TIER IV you must provide fail-safe cooling and so the cooling must use a UPS. You have two options – either use the UPS that are for servers OR put your own UPS for cooling. We consider this second solution to be better and safer and therefore our cooling in each datacenter will have 2 separate UPS, each of these UPS must be able to provide 100% power for 100% cooling of the entire datacenter.
Separate UPS are for offices. Again, we keep them separate from the UPS that powers the servers.
All UPSs must be designed for a minimum of 15 minutes of backup to 100% of datacenter capacity. If you take into account that everything is doubled, you are guaranteed 30 minutes via UPS. In our case it will be even more, because we will have a UPS in 2N + 1 mode. We will always have one extra UPS on each branch and it will serve as a backup in case of failure of another UPS.
As with the motor generators, we decided to buy the UPSs from different manufacturers on different branches at different times to be safe. It’s a complication with servicing, but it’s safer because we won’t be hit by some hidden serial defect in the future.
IT elements
All IT equipment must have dual power supplies (at least two power supplies) or be connected via an ATS to ensure that both power supplies are used. In this case, this is a slightly different ATS than the one we have in our scheme and discussed above. These rackmount ATS work similarly, but they are “electronic devices” that switch small currents (usually up to 16A). Everything is done electronically. The ATS we are referring to switch currents of 630 amps or more) and are switched by powerful electric motors. Just for completeness, it takes from 10-100 ms to switch from one power supply to another and most power supplies for servers do not recognize such fast switching.
Want to see our power supply diagram?
We’ll probably give away the most, but we’ll show you our power schematics. Such schemes are usually secret due to the fact that they are labor intensive and considered trade secrets. Due to our openness, we will show you the diagrams.
What can you look forward to next?
Next time we will write down more information and details. We’ll also reveal some details from behind the scenes. Finally, you will also learn about the most common errors or the economics of operating such a datacenter. The economics of the operation will be on a separate chapter.
Since we enjoy TIER IV certification and take it as one of our biggest challenges, next time we will discuss in detail some of the technical prerequisites that a TIER IV datacenter must meet.
He will surely get to real examples that will help you understand why no datacenter in the Czech Republic (or elsewhere in Central Europe – including Germany, Poland, Hungary, Slovakia, Austria) has TIER IV certification yet. If you have any questions, please write to us and we can take the answers into account in future articles.
Why are we doing all this?
There are several reasons. We are the largest hosting company in the Czech Republic, we host the most services in our datacenter in the Czech Republic and so we are aware that we must aim everything to the maximum satisfaction of our clients. The quality of the datacenter is clearly a very important factor for our further growth and development and without a quality background it will definitely not work. )
That’s why we want to have 2 modern datacentres that will meet the most demanding criteria and we want to have the whole thing certified in this way. We are not going down the road of someone drawing something somewhere and someone somewhere quickly building it. The way we go about it is that we spend a huge amount of time preparing, we’ve been preparing for several years with a team of several people. We keep an eye on everything on the construction site and make decisions promptly. As a result, we want to handle all operational matters “in house”. For example, we have our own electricians (even 2). So we know everything in our building. We can figure it all out on our own. As quickly and as best as possible.
Both datacentres will appear as one on the outside, but in reality they will be two separate buildings that will be able to operate completely separately or complement each other in operation. It’s going to be like a RAID datacenter.
Yes, none of this makes sense economically, but we do everything we can to ensure maximum satisfaction for our clients. That is the primary objective. We do it because we enjoy it. The economic side is a secondary issue and as we say: “If the clients are happy, the profits will come”. We are a strong company that runs without credit or debt. We own the datacenter, the infrastructure and we don’t have to “report to a bank”. We are building a second datacentre for tens of millions of crowns, again without any help from third parties. Maybe you could keep the money and buy something “for fun”, but we enjoy this and we enjoy happy clients.
The planned certifications should be a guarantee to our clients that their data is in the best place and well taken care of.
Stay tuned to see how our plans are coming along.