As you’ve probably noticed from previous blogposts, we’re working on several projects right now. Let’s start with a quick summary and get right into the topic in the title 😉
What we are currently working on
Our technicians and developers are working on WMS, WEDOS Cloud and one project that we will introduce soon 😉
The sales department is catching up on several advertising campaigns and is already negotiating with the first VIP clients. We are also finalizing the distribution of WEDOS Energy in cooperation with a company that has huge warehouses and the capacity to ship tens of thousands of energy drinks every month.
And then we have our colleagues from datacenter management. They, together with our accredited TIER specialist, are currently working on answers for the Uptime Institute regarding the completion of the official project documentation. Everything is on track, it’s just a bit more challenging as we want TIER IV. This month they completed testing of another air conditioner for DC1. Now they plan to complete the technology in DC2 and a second upgrade of the second branch of the power supply in DC1.
They are also in charge of the oil bath project for DC2, where all the servers will be cooled in an oil bath and the heat will be dissipated into the nearby city pool. Today we would like to introduce you to the Oil Bath v4.3 🙂
Oil cooling
The principle of cooling computers with oil is not complicated. You just need oil that doesn’t conduct electricity. Ideally one that doesn’t spit and doesn’t smell 🙂 Then just passive radiators and water cooling to take the heat out of them. You can find plenty of examples of such PCs on YouTube.
However, it is more complicated with servers. They are hardware and software built for air cooling. They are protected at the firmware level against overheating, fan failure, and have physical components that help with airflow cooling but hinder oil cooling.
Next, you need to resolve redundancy for the servers. It’s not a home PC where the failure of one component means the end. Servers must be able to function if a component fails, because that would mean failure.
While with air cooling you don’t have to deal with this, or the redundancy is at the datacenter level (additional air conditioning), with oil cooling it has to be dealt with by another circuit. These are additional pipes that must be run separately for TIER IV, similar to electricity or connectivity (one branch through the ceiling, the other through the floor).
Oil bath v4.3
The problem is also the tub itself, where the servers will be placed. In our case, we already use HPE Moonshot server enclosures, which Hewlett-Packard developed for large-scale cloud solutions. Each 4 1/3 U HPE Moonshot enclosure contains 45 servers and 2 Switches each 4x 40 Gbps out. Inside, the individual servers are connected 2x 10 Gbps. Just the solution for a big and fast cloud.
We have selected and tested this solution with oil cooling in mind. When we put the HPE Moonshot in the tub, the technicians pull all the components out from the top. We don’t mind the powerful HPE Moonshot servers we’ve chosen to put in them are incredibly warm. The oil will keep them cool without any problems. Compared to the outdated air cooling, oil is somewhere else.
When you invent something like this from scratch, you run into several problems. Some of them you will find out when you discuss them in a team, others will be revealed by professionals in different fields and others by engineers who produce similar things.
Over the years of testing, many great ideas have also emerged that will greatly simplify maintenance and increase efficiency in the future. We’ve slowly been somewhere else since the original concept, and we dare say that nobody else in the world who plays with oil cooling does it anymore, or even similarly 🙂
Of course, everything can be made, but in the end it’s all about the price. We can’t pay more for the tubs than we can for the whole of DC2. And where are the problems?
- Baths must be made of commonly used material. Everything is possible today, but the ideal material would have to be custom-made.
- Baths shall be manufactured using commonly used technology. We wanted a local manufacturer, not importing them from abroad.
- The baths must be manufactured with technologies and materials with which our supplier has practical experience. We don’t want to spend the next few years testing.
In order to achieve this, we had to step back from the requirement that we wanted tubs that could fit four HPE Moonshots at once. Instead, we’ll just put two in each.
On the other hand, we have managed to develop a unique variant that does not require ribbing for water cooling and yet maintains redundancy.
The first Oil Tub v4.3 is currently under construction and is also a hot candidate for our data warehouse in DC2. Of course, further modifications are still planned. Of course, if everything goes according to plan, then only cosmetic.
Oil tubs come on these heavy duty racks in our data warehouse in DC2.
