We have done a number of experiments with oil cooling servers. Some took literally a few minutes and other times the whole project took several years to collect enough data for long-term operation. As a result of a series of experiments, our second datacenter will house all of the planned 10,800 physical servers cooled in oil baths using a giant cooling circuit. It includes a project to prove (proof of concept) that waste heat from oil cooling can be used to heat remote buildings. Of course, we don’t stop there, we still have the HUPS project 😉
History of WEDOS oil cooling
We started experimenting with oil cooling at WEDOS in 2013. We knew it could work, even in many ways surpass the physical limits of conventional air cooling. On the other hand, we wondered, if it is so great and cheap, why doesn’t anyone else use it commercially?
Year 2013
Rather as a joke, we had a custom plastic tub made and put an older but functional server in it. You can see the server flooding and our immediate reaction in the video 🙂
As part of this project, the server was supposed to be sunk for a week to see how it would turn out. Eventually, it dragged on until 2015, when we took the project further.
We needed to investigate the condition of a server that had been in regular transformer oil for two years. In fact, it was as good as new. It was not even close to the servers that are air cooled in the server room. We understand that the oil will also have a very positive effect on the life of all components.
For the next project, we started to deal with the impregnation of hardware with the possibility of subsequent degreasing, which could be useful for claims.
Year 2014 – 2015
The concept of oil cooling made more and more sense to us. So we started looking around the world for real installations. They had one of these at the University of Vienna, where it served as a demonstration that it was technically possible. Our boss went there for a look and on his return gave the green light to the next phase of the oil cooling project.
We bought a half-rack, which we completely disassembled. We only needed a skeleton of him in the end.
We had a custom-made tub made of hardened plastic and then a catch tub in case of an accident.
Furthermore, it was necessary to create a real water cooling circuit with heat exchangers, pumps, fully backed up in case of failure.
It also took many months to find the right oil. The conventional transformer oil was extremely greasy and we were also concerned whether it was too aggressive. After a lot of research we finally decided on pharmaceutical oil, which had very good properties – it is not harmful to health, it is practically non-flammable (although it is listed as class IV, so you can’t burn it with direct fire), it doesn’t smell, it conducts heat well and it has a lifespan of 30 – 50 years.
For our experimentation, we ordered 1,000 liters just in case 🙂
We built and prepared the entire cooling system for the 2015 Open Days, when we planned to sink test servers, including one brand new one from the product line designed for the then NoLimit and VPS SSD services.
Of course we have video of the server drowning 🙂
We could monitor and control the entire cooling system remotely. The developers have programmed a complete system for this.
We have conducted dozens of tests and experiments. We monitored both hardware and software/firmware behavior. Over the course of several months, we have gathered a large amount of data, experience and knowledge that has determined the direction and development of oil cooling in our country.
Many of the findings were quite revolutionary for us. We’ve opened up a whole new “dimension” where things work differently, you just have to figure out how to take advantage of it. However, this will be more for future projects 😉
Year 2016
In 2016, the construction of our second datacenter was already underway, and it was already decided that all servers would be cooled in oil baths. From them, heat will be transferred to the nearby municipal swimming pool via a water circuit via heat exchangers.
Due to the market situation, or where the development of servers and software is heading, we have switched from the concept of large tubs for individual servers to tubs for server cabinets. Nowadays you don’t scale by upgrading one server, but by adding more servers. Individual servers have their physical limits and in addition, the price increases significantly from certain parameters, while if you can spread the load on more machines, you scale significantly cheaper and you will not encounter physical limits of hardware.
For this need, we switched from deep tubs with a rack structure inside to basically a regular tub where you can just put the server cabinet. Plastic replaced stainless mainly for safety reasons – it doesn’t burn 🙂
Next, it was decided that we would bet on HPE Moonshot server cabinets, which allow us to concentrate huge computing power in a very small space, but at the cost of needing extreme cooling, which we didn’t mind. That’s what we have oil for. Each HPE Moonshot fits 45 physical servers and has 2 switches with uplinks up to 320 Gbps. Price … like a family house 🙂
Each HPE Moonshot will have to be disassembled and impregnated. This ensures better thermal conductivity and easier degreasing.
We also tried degreasing. There are means for this that, in combination with a suitable impregnation, completely remove the oil from the hardware.
The result was the following 2 oil cooling prototype, designed for long-term testing of the HPE Moonshot.
Of course, we also have video of the oiling again 🙂
2017 – 2020
This was followed by long-term tests, consultations with manufacturers of hardware, cooling, pipes, heat exchangers, pumps … We even had several projects running in oil for which we provided free hardware.
It was important for us to confirm what we already knew from previous experiments and to prepare the ground for the oil baths in the second datacenter.
Thus, most of the research and work related to oil cooling has been in the context of baths.
In 2019 we already had a hot candidate see. article How the development of oil cooling continues: oil bath v4.3. We even had her right here in DC2 for her first stress test.
Only a few adjustments were needed to make the servers easier to handle.
Eventually, however, more fundamental changes were made and the result was version 5.1, which had one serious design flaw.
So currently the server oil cooling project is waiting for a new bath. Of course, it’s not the only project related to oil cooling 😉
HUPS project
After successfully “drowning the servers”, we realized that oil cooling has great potential for the future beyond servers. That’s why we started thinking about what else would be worth drowning.
And that’s how the HUPS project (Economical UPS) was born. Yes, that’s right. We’ve been preparing to drown UPS for months. We have a disassembled UPS in the DC2 server room and were waiting for a suitable receptacle, which was delivered last week.
It’s going to be a whole new challenge. What exactly are we planning? We’re considering doing an online broadcast of the UPS dive. You won’t find anything similar about UPS diving into oil anywhere, so it will probably be a world premiere.
We are in contact with a manufacturer in UPS from France and Italy who are looking forward to our experiment. First we want a clean sinking of the UPS in oil and we hope. In the first step, we do not plan firmware modifications and technical changes that we have with servers. Yes, there we have for example modified firmware from HPE manufacturer directly from developers from USA. Don’t expect that with UPS 🙂
First we would like to drown a small UPS 30KVA from Socomec. If everything will be fine, we are ready to drown 4 more UPS and each of 80KVA.
We’re thinking of streaming everything online. So you have something to look forward to. We are ready for it to fail 🙂