WAF report from WEDOS Global Protection for July 2023

[gtranslate]

Even during the holidays we did not slack off and continued to build our WEDOS Global infrastructure. We optimized routing in selected locations, continued to arrange new ones and are finishing up some broken ones. As a result, we’re getting better response rates in several countries, and we’re also testing several new features that we’re going to make available to all users. Of course, there were also DDoS attacks, although in terms of number and strength it was more like a holiday traffic.

WEDOS Global

WEDOS Global is the name of our global infrastructure that offers automated cybersecurity solutions, web speed enhancement and integration of modern internet technologies.

It is built on a large number of our own physical servers (currently over 1500) in 25 locations around the world, with connectivity currently exceeding 2500 GB/s. Everything is built on BGP Anycast and reverse proxy technologies. providing WEDOS Global users with a fast, stable and secure connection.

The WEDOS Global Protection service we run on it, which you can already purchase and use, acts as a reverse proxy. It optimises speed, filters threats and ensures that websites are accessible and loaded quickly from any part of the world. By connecting to local and global network nodes (IXPs) and using modern technologies such as HTTP/3 or IPv6, WEDOS Global ensures that websites are not only fast, accessible and secure, but also easy to deploy and maintain technologies that are not always simple to get up and running.

WEDOS Global Anycast DNS

For comparison with competitors, we use paid third-party measurement. This is primarily used to tune the speed in each state (we know where we have weaknesses, where to put the next point, where to look for peering, etc.)

We are currently:

• TOP 25 in the world
• TOP 10 in Europe
• TOP 30 in North America (this is an extremely competitive environment)
• We made it into the top 20 in the Australia and Oceania region
• We made the top 20 in Asia (this region is extremely difficult because many countries have specific laws, ISPs often don’t “talk” to each other, etc.)
• We are in the TOP 20 in Africa

Other points we are working on

We did not launch a new point in July. We already have the “easy” sites, now it is the turn of the more difficult ones, where it often depends on legislation and guarantees.

As we take the service further and attract new demanding customers, we are also discovering new interesting places where we could build more points in the future.

Ireland (Dublin)

We want to strengthen our services in the EU. Ireland, specifically Dublin, seems like an interesting choice. We already have a location. Servers, switches and other hardware are already packed in our second data centre WEDOS DC2. However, we lack to arrange connectivity. Once you want a guaranteed 100 Gbps somewhere with the possibility of further increases, it’s slow going.

If everything goes according to plan, we could launch the site in Ireland by the end of the holiday season.

São Paulo (Brazil)

South America is the last permanently inhabited continent that we lack. However, finding a suitable location is quite challenging. Negotiations have been dragging on for over six months. So far we have only managed to reach Brazil (São Paulo), where the servers are currently waiting for clearance. However, after the experience in Mexico and Turkey, we prefer not to give any or an approximate launch date.

Frankfurt am Main (Germany)

You may be thinking that Germany is a short distance away and the few dozen ms are not worth it anymore. However, it is not that simple. WEDOS Global need to have locations at all major crossroads of the Internet. And one of the most important ones is in Germany. Moreover, if we want to be in the top 5 in Europe, you can’t do it without a location and connections in Germany.

This month we signed the necessary contracts. Now we have to prepare servers, switches and other hardware. Arrange transport etc.

Other planned WEDOS Global locations

We would like to have a site in Dubai and Oman in the first phase, but there are problems with legislation and price. India is also very attractive, but we face the same problems here and we would have to set up a branch there. We’ll deal with everything in time.

We are also looking at locations for service improvements in the EU – mainly Hamburg and Rome. Then it’s about arranging connections. In the next phase we will add locations in Hungary, Austria, Spain and we are also considering Slovakia. However, it is definitely more the demand from customers in these countries. If there is a significant increase in demand for our services somewhere, we will set up a location there relatively quickly.

New peering

We expanded our network infrastructure in Silicon Valley and Chicago. This is through new connections to IXP (Internet Exchange Point) and ECX (Equinix Cloud Exchange).

The new links will be built on 10G ports, which means they will be able to transmit data at 10 gigabits per second. Several of these 10G ports will be commissioned in each of the locations. Further performance and throughput improvements will be achieved through the use of Link Aggregation (LAG) technology, which will allow two 10G physical ports to be combined into a single logical port with a total capacity of 20G. This move will lead to an increase in available bandwidth and increased redundancy.

Both sites have a high concentration of datacentres and network equipment. This makes them hubs of internet traffic and important nodes through which large amounts of data pass. All the major companies (such as Apple, Google, Facebook and many others) will be closer to us this way and the traffic from them (especially their bots) is really not negligible.

Would you like to learn more about WEDOS Global?

If you are interested in WEDOS Global and would like to learn more about the advanced technologies we use, for a deeper and more detailed look into the technology architecture on which the WEDOS Global infrastructure is built, we recommend you listen to our presentation from Kubernetes Community Days Czech & Slovak 2023. This expert presentation was led by two colleagues who play a key role in the development of WEDOS Global.

WEDOS Global Protection

WEDOS Global Protection is the first service launched on the WEDOS Global infrastructure. It was originally intended to serve as protection against various types of DDoS attacks, but it turns out that content caching works perfectly for some attacks. We have moved it a bit and WEDOS Global Protection now works as a web CDN. So, under ideal conditions, it can be used to stop a very, very large influx of regular visitors.

This was ultimately the reason why we dropped the free plan for personal non-commercial websites. It can already do so much and its contribution is so great that we believe it is always worth paying 25 CZK per month.

Testing HTTP/3

In June we started testing the possibility of deploying HTTP/3 for WEDOS Global Protection users. In July, we moved on and our website is running completely on HTTP/3. If we don’t encounter any problems during testing, we will run HTTP/3 for everyone who uses WEDOS Global Protection (WGP can be used even if you have hosting with a competitor).

And why should you care about HTTP/3?

HTTP/3 is the latest version of the HTTP protocol, which is the basis for data transfer on the Internet and allows you to access websites.

Earlier versions of HTTP (HTTP/1 and HTTP/2) use the Transmission Control Protocol (TCP) to transfer packets. TCP is reliable because it ensures that all packets arrive at the right place and in the right order. But it also has drawbacks – if one packet is lost or delayed, all subsequent packets must wait until the lost packet reappears. This is called Head-of-line blocking.

HTTP/3 instead uses a newer protocol called QUIC, which was designed to overcome the head-of-line blocking problem. QUIC uses User Datagram Protocol (UDP), which does not require packets to arrive in any particular order. This means that if one packet is lost or delayed, other packets can continue without waiting.

The benefits of HTTP/3 include:

• Better performance on unstable networks: HTTP/3 can transfer data more efficiently on networks that are unstable or have high response times, such as the mobile Internet. For customers with mobile phones in poorly covered areas, on the train, etc. you will gain a competitive advantage.
• Faster connections: the QUIC allows for faster setup of new connections, as fewer steps are required to set up a connection between the client and server.
• Security is the default: QUIC uses encryption as standard, which improves data security.
• No Head-of-line blocking: the QUIC allows packets to continue without having to wait for lost or delayed packets. This improves the speed and efficiency of data transfer.

With WEDOS Global Protection, your website will not only be protected, but also faster thanks to the WEDOS Global network (Web CDN, Anycast DNS). Like HTTP/3, IPv6 or HTTPS can be used even if your hosting provider does not support it. More technologies are in the pipeline.

Own certificates for WEDOS Global Protection

The ability to use a custom SSL/TLS certificate in WEDOS Global Protection was another task we had on our to-do list. In July, we carried out the last testing and soon it will appear in the administration for Expert and Ultimate plans.

Statistician WEDOS Global Protection

The stats were mainly affected by the less active forwards in July. Tens of millions of requests in the form of L7 HTTP flood attack have never been tried. It’s just a waste of resources for the attackers. They try a few hundred IP addresses and a synchronized attack to cause a problem before specific protection kicks in, limits are applied, local blocking is turned on, etc. Usually it’s a few seconds, then they shut it down. Large attacks are thus rather the exception and the new L7 HTTP flood standard is a short attack in the lower units of minutes.

We’re also stricter on traffic from VPN services that are free or offer free periods. Virtually only malicious traffic comes from the IP addresses of these services. In the future, accesses from these IP addresses will always have to pass a redirect test.

The following statistics are from the reverse proxies at each point that handle requests cleared of L3/L4 DDoS attacks.

Furthermore, some of the requirements that we use to monitor the availability and performance of individual points are not included in the overview so as not to bias the statistics.

At the end of July, the number of domains protected by WEDOS Global Protection increased to 4278 domains (+524). Some of these are domains that have been added by support due to attacks, or they’re heavyweight sites that are greatly helped by automatic caching on the proxy server. Outside of our country, 1029 users (+89) use WEDOS Global Protection.

In May, a total of 3,568,662,296 requests (+5.34%) were recorded out of 8,641,401 (+11.79%) unique IP addresses that were directed to protected domains. On average, the proxy servers handled 115,118,138 requests per day.

As far as the increase in unique IP addresses is concerned, seasonality has an impact. The majority of WGP users are still from the Czech Republic and Slovakia. Their visitors were more likely to join from abroad. There is also an increase in large customers from abroad. We also have websites of ministries (not the Czech Republic, they don’t even want to talk to us about our protection). That’s a different composition of traffic.

As we improve the algorithms that temporarily blacklist IP addresses, the number of requests we have to block to the reverse proxy (access limits, connection limits, and web firewall) decreases.

Bored protection stats 🙂

• L7 DDoS – intercepted by limiting accesses (HTTP flood):
  1 667 752
• L7 DDoS – intercepted problem connections (Slowloris, Connection Exhaustion, etc.):
  1 657 563
• Blocked by WAF rule: 10 662 108
• Further blocking of L7: 5 581 467

These numbers are just first attempts. Once there are repeated attempts that gain momentum (perhaps tens of thousands of problematic attempts per minute), the IP address goes blacklisted. But it’s more complicated than that because we treat different IPs differently. Likewise about different forms of attacks.

The next report will have higher statistics. After the test run, we will deploy protection to all protected WordPress sites on the login form. In recent weeks we have seen an extreme onslaught of brute force password cracking attacks. Attackers are doing a lot of inappropriate things and are hitting forms in bulk, which leads to exhaustion of allocated server resources. If you have experienced intermittent unavailability or slowdowns of WordPress in recent weeks, this may be the reason.

We will solve it so that if someone wants to log in, they will have to go through a redirect or captcha in case of a suspicious IP.

L3/L4 DDoS attacks

Actually, we don’t encounter many L3/L4 attacks in the last few months. We used to experience 10 Gbps+ attacks several times a week, the last one was in June.

In July, there were 6192 of them, but the strongest one was 7.7 Gbps and 2.9 million packets per second at peak and was directed to one of our VPS customers.

The most powerful L7 DDoS attacks

Two things characterised July. Increased attack activity from VPNs. These attacks range from L7 HTTP flood, to finding vulnerabilities in popular content management systems, to SQLi attacks. Declaring that free VPNs are in fact hidden botnets in July and even in August is certainly not far from the truth.

Another interesting fact was the increased number of attacks directed at gaming websites and servers. Some were quite strong. This type of site comes under attack regularly, but more than usual in July. If you run a similar project, definitely consider WEDOS Global protection – see. WEDOS Global Protection for Game Servers, Gaming and eSports platforms.

1. Attacking a game server website with over 2M requests per minute

This was the strongest attack in terms of the number of requests per minute at peak times. According to the logs, it was over 2,041,522 per minute. In total, nearly 4M requests from 2693 unique IP addresses went to the target. Attackers know that WGP will quickly start blocking these attacks and the most active IPs will end up on the blacklist for good measure, as can be seen in the graph. Interestingly, this time they were impatient and tried again in less than 10 minutes, but the most active IP addresses that no one would miss were already on the blacklist and WAF took care of the rest.

2. Attacking a game server website with over 1M requests per minute

The next attack was a real blitz. It lasted a little over a minute. A total of 1.25M requests from 1616 UIPs.

This attack was interesting because the actual brute force, it went through the Singapore and Dallas location. Any centralized attacks can be dealt with far more effectively by a given location without affecting the rest of the world. For example, in case of emergency, we could run a javascript redirect or captcha check for everyone (outside of whitelist) on that site.

3rd place attack on a game server website over 633 thousand requests per minute

Another game server site came under attack in the second half of July. Again a very short attack of just over a minute. 862 thousand requests from 692 unique IP addresses. At peak, “only” 633,468 per minute. Here, it looks like the global synchronization of the attack completely failed, so the protections had it easy.

Conclusion

Anyone can use WEDOS Global Protection to load faster and protect their websites. The service can be used without the need to move hardware or change web hosting providers. Simply point the domain to the WEDOS Global DNS and add the domain to the WEDOS Global Protection administration.