There’s a lot going on here. We are building our decentralised WEDOS Global network, preparing to launch the first WEDOS Global Protection service on it, and the development of oil cooling is still going ahead, but these are not all big projects. Last year, we migrated our service data to the new 3PAR StoreServ Storage 8450, which was very challenging and we still ran into the occasional problem a few months later. In addition to significant simplification, increased stability and speed, this new solution also opens the way for us to take many services further.
Now you can also listen to the article as a Podcast.
HPE 3PAR StoreServ Storage 8450
Professional AllFlash (100% SSD) storage that has full component redundancy and all the advanced features you expect from modern datacenter storage.
For us, it was essential to choose a solution that would work for all of our hosting services. If you have one kind of hardware that works well for web hosting, VPS and Cloud, you will not have to worry about much. We also save the cost and space of having backup equipment that you always have to keep in stock should it be needed (breakdown, increased demand, supply problem, etc.).
3PAR pleasantly surprised us. It’s ALL FLASH = full SSD (NVMe) storage, where each array in the configuration we use can handle almost a million IOPS of real traffic (no benchmarks, but long-term real-load testing). There are fast drives that can do many times more, but in real life you run into other limits (controllers and different paths). The configuration of each 3PAR is at least 2 controllers online and 2 independent paths from each. Of course, everything is serviceable on the fly (i.e. controllers). Even each disk is connected one way to one controller and another way to the other.
We currently have a total of 12 3PAR StoreServ Storage 8450 in operation in the WEDOS DC1 datacenter. Then we have another one to test.
Simultaneous storage of data in two or more copies
One of our goals is to make the data repositories redundant. If one goes out, so that traffic can come from the other. We have managed to solve this, but it is not automated yet. For full automation, you need to test a number of complex scenarios. It’s not enough to prepare the system for one thing to fail, you have to expect more to go wrong at the same time.
And how have we resolved this so far? The first 3PAR StoreServ Storage 8450 stores and reads data. The second backup 3 PAR just stores the data and takes care of creating snaphosts (see below).
We work with live data in real time on two 3PARs. They are currently in our first datacenter WEDOS DC1. After the holidays, the second of the pair will be moving to the second WEDOS DC2 datacenter.
In addition, we are preparing a third storage (remote backup) that will not contain live data, only a copy of it. We would like to store it outside our datacentres somewhere in the Czech Republic. The plan is for this backup to be protected against encryption (ransomware attack) or other forms of data corruption. This 3PAR should download and store the data itself. Stored data will be in read-only mode with periodic purging of old data. This will be our insurance policy if all else fails. We simply won’t lose customer data 🙂
Snapshots
Imagine being able to make a real-time copy of your computer’s current state (the state of data on your hard drive and in memory) and save everything. You can return to this state whenever you need to. This is called a snapshot. It’s like saving a played computer game before an important fight 🙂
Snapshots are mainly used for virtual servers. If you plan to make a major system intervention or update something, you save a snapshot and if it fails, you upload it back. It’s faster than restoring from a backup. If you use a fast SSD and the virtual server is not very large, it can be a matter of seconds.
With the switch to the 3PAR StoreServ Storage 8450, we thought we could back up our data this way and restore it very quickly in the event of a problem.
Thus, we perform the following data backups on selected 3PAR repositories:
- Snapshot is taken every 4 hours and held for 24 hours (6 backups in total)
- Snapshot is taken at night, every 24 hours and kept for 7 days (7 backups in total)
- Snapshot is taken every weekend and we keep them for 4 weeks (4 backups in total)
- Theoretically, we are now thinking of holding a deposit 3 to 6 months in arrears
The frequencies and times are only for testing purposes. We draw on our experience. The 4 hour ones are mainly for us, if something breaks a lot, so that we can get everything up and running as soon as possible. The daily ones are in case the data is corrupted. Weekly solves the problem of deleted or unsaved data.
If we offer the service of such backups to customers, it is possible to keep backups for a longer period of time. Feel free to do 1x per month and keep it for 6 months, but there would already have to be a customer request due to GDRP.
We test snapshots for almost all services – web hosting, WebSite, WMS, VPS ON and WEDOS Cloud.
Transition to automation
Once we’ve tested everything properly and prepared for different scenarios, we’ll start working on automating and restoring data from backups. Since they’re snaphosts, it simplifies a lot of things. The service will be restored to the way it was on a certain date.
The recovery process will be automated. A backup from the second or third 3PAR will be requested, prepared and restored. Depending on the amount of data, this can take seconds to minutes from the second 3PAR and tens of minutes to hours from the third 3PAR (remote storage). In theory, we will also be able to provide a downloadable backup (upload to another repository).
Conclusion
Our goal is to be able to offer a frequency of backups, deployment and security that meets all the standards that even the most demanding customers may require.
It will be a few months before we get to that point, but we are making great progress. There have been some problems associated with that, which we have encountered from time to time, and there have even been a few outages because of that, but if you want to take the service to a whole new level and offer it to the general public, you can’t do it without that. Those who do nothing do nothing, but in time they will find that they have slept through progress. We are not afraid of new challenges and we are moving towards progress 😉