Building WEDOS Global – New points, centralized sheets and other moonshots

[gtranslate]

Building a decentralised global network that not only protects us but also allows us to provide entirely new services on it is not only expensive but also quite organisationally and bureaucratically demanding. Definitely one of the biggest challenges we have faced so far.

Now you can also listen to the article as a Podcast.

SpotifyApple PodcastsGoogle Podcasts

New HPE Moonshot 1500

Each WEDOS Global point will house an HPE Moonshot 1500, which is a server enclosure with 45 physical servers and 2 switches. HPE has developed Moonshots for demanding cloud solutions where you need to scale your infrastructure quickly and easily. Fully loaded is not exactly cheap or lightweight. They weigh over 80 kg. The size of the server is 4 and 1/3U, in which you can put more servers than in one regular rack, which is also completely wired.

Each HPE Moonshot has 4x 40GE and 8x 10 GE ports with full duplex connectivity, so it can theoretically handle up to 320 Gbps of network traffic. However, our main concern is that we can have a lot of connections to local providers in each location. We will address connectivity through 40 GE and peering through 10 GE.

We want to have at least one HPE Moonshot at each point. If we need to scale, we’ll just send another one, but it’s better to add another location. When we have dozens or hundreds of sites, we will reinforce the existing – already existing – points.

In the first phase, we want to have a total of about 45 “boxes” of HPE Moonshots in strategic locations around the world. In March 2022, we had 18 units in stock for this purpose, some of which we have already shipped. Read more in the article How the WEDOS Global build is going, oil cooling and other April news at WEDOS.

In May, we received another batch of 21 pieces of fully equipped HPE Moonshot on three pallets with a total weight of 1844 Kg. After unpacking, you will have to thoroughly test and prepare the software. We place cables and other accessories according to the requirements of the datacenters where Moonshot is going.

We already had several other boxes in operation from our own stock. So we have prepared HW for almost 50 locations without any problems.

We expect these new ones to go mainly to more distant locations (North and South America, Asia, Africa and Australia).

New and active sites

Currently, 9 HPE Moonshots are already in place and operational. Now London is getting involved.

Active and in use (9)

🇨🇿 Hluboká nad Vltavou – currently dealing with Czech and Slovak traffic.
🇨🇿 Prague – in charge of selected EU traffic
🇦🇹 Vienna – traffic from south-eastern and southern Europe and some traffic from the east
🇳🇱 Amsterdam – Western and Northern Europe and the rest of the world including the USA and some traffic from Russia
🇸🇪 Stockholm – traffic from the Nordic countries, the Baltic States and Russia
🇫🇮 Helsinki – traffic from the Nordic countries, the Baltic States and Russia
🇨🇭 Zurich – handles traffic from Switzerland, Italy and a chunk of Africa
🇫🇷 Paris – handles traffic from France, Portugal, Spain and a small chunk of Africa and America
🇵🇱 Warsaw – handles traffic from Poland and a small part of the Baltic countries

Individual sites are already clearing traffic. Maybe this is what the morning of the 8th looked like. June 2022. The spikes are the attacks that WEDOS Global Protection will not let go any further.

Minute traffic on WEDOS Global in the morning 8. June 2022
Minute traffic on WEDOS Global in the morning 8. June 2022

On site but not in use (1)

🇬🇧 London – getting involved.

Arranging details/sending ()

🇰🇷 Seoul – agreed location, we are solving peering
🇹🇷 Istanbul – we are arranging the details, everything is under negotiation
🇧🇬 Sofia – we are arranging the details, everything is under negotiation
🇪🇸 Barcelona – negotiations underway, all in talks
🇸🇬Singapur – we are arranging details, everything is under negotiation
🇯🇵Tokio – we are arranging details, everything is under negotiation
🇨🇳Hong Kong – we are arranging the details, everything is under negotiation
🇦🇺Sydney – we are arranging details, everything is under negotiation
🇷🇴Bukurešť – we are arranging details, everything is under negotiation

We are also negotiating with about 30 other sites. Some things work and some things don’t. We have a team of people who are also looking for potential locations with good connectivity and reaching out to datacenters.

We also reached out to datacenters in North and South America.

At the same time, we are trying to create processes so that everything can be accelerated and scaled in the future. However, we are encountering really extreme differences between individual datacentres, both in communication and technical service, and we are only in Europe.

Current status of the WEDOS Global network

As we launch individual points across Europe, we’re also testing how things are accelerating. It really makes a huge difference when you get a direct connection (peering) to the networks of the biggest providers. Already it appears that we could achieve a 5 ms response time almost all over the EU in the near future.

For now, the acceleration is as follows. This is an indicative one-off measurement by a third party in ms. As the points accumulate and we gradually start connecting with local providers in the localities, it will get a lot better 🙂

LocationAnycast DNS
WEDOS Global DNS
Unicast DNS
WEDOS DNS
Acceleration%
Amsterdam1221942,86 %
Berlin13281553,57 %
Brussels3181583,33 %
Dublin16341852,94 %
Edinburgh17452862,22 %
Eindhoven2252392,00 %
Frankfurt9900,00 %
Gothenburg12311961,29 %
Haarlem2191789,47 %
Hamburg15281346,43 %
Lelystad9201155,00 %
Lille3236411,11 %
Leipzig181800,00 %
London8231565,22 %
Milan4252184,00 %
Oslo12271555,56 %
Paris2272592,59 %
Prague022100,00 %
Rome4231982,61 %
Salzburg413969,23 %
Demonstration of response speedup using WEDOS Global DNS

WEDOS Global for everyone

Everyone will need WEDOS Global and especially WEDOS Global Protection. Performing successful DDoS attacks through the application layer (L7) is relatively easy today. Because of the war in Ukraine, a number of groups have emerged from Russia that provide free scripts to carry out these attacks, including instructions on how to use them. Now we don’t mean sites where there is JavaScript attacking disinfo sites.

We are referring to Python scripts that perform real attacks via proxy servers and it should be noted that they are quite effective. You will remember the attacks in April, which limited the functioning of a number of websites in the Czech Republic, including NUCIB. We went through these scripts, found out how they work and how to prepare WEDOS Global Protection for them.

So many of these scripts are being played with by a number of “upstarts”. Last time we recorded an attack on an elementary school website, someone didn’t want to take a test 🙂

An attack on an elementary school website.

Unfortunately, the customer was not using our DNS, so we couldn’t quickly hide it behind WEDOS Global Protection and had to deal with it the old-fashioned way, which took a long time. As you can see, the attacker was able to generate an attack with the strength of over 68 thousand requests per minute, which passed to the webserver. And much of it was blocked by blacklist filters. The overall strength of the attack was thus higher.

If an elementary school student can pull off such an attack, we have a lot to look forward to 🙂

But how do you explain to the school principal that you need WEDOS Global, a decentralized global network built on BGP anycast providing a reverse proxy that will protect the web in different ways based on URLs, source IP addresses, useragent, referrer and other request header data?

So my colleague came up with the idea of an educational video, where he would present the concept of WEDOS Global and the related services on something simple and real. Try looking.

Centralized lists not only for WEDOS Global Protection

Our team of developers has grown a lot in the last six months, so it’s time for some resty. This includes centralized lists of IP addresses that are or may be problematic. We pay for blacklists with attacking IP addresses, we also use various freely distributed ones and last but not least we create our own lists based on traffic analysis from web hosts, WMS and WebSite.

Until now, however, each department has worked with them in its own way, or even had its own. Some are relics from ancient times. We decided to change this and create central lists where each department can quickly access. These lists will be centrally updated and maintained. These are not only blacklists and whitelists, but also lists of search engine robots, social networks, accessibility monitoring services, APIs of various services used, etc.

Firstly, the existence of these lists will ensure that someone, for example, does not forget to whitelist the APIs of the plugins used for WordPress, and also if we need to add an IP address, it will be done centrally and will be reflected on all services and parts of the network infrastructure.

In the future, we are considering offering these lists. We believe there would be interest in them.

Conclusion

Building WEDOS Global is expensive and challenging, but there is no other way. Without WEDOS Global, we cannot build WEDOS Global Protection, which will be a necessity for most sites in the future. We still plan to have a free tariff for personal use and a very cheap tariff for businesses.

If you want to help us with this, we are always looking for more and more colleagues. We have plenty of work and you can grow professionally and knowledgeably very quickly 🙂