PostgreSQL locks on an nginx server of mine were through the roof and load had reached 55.0, yet the host only had eight CPUs. Was there some problem, or had a link to our website landed on the front page of the New York Times? A quick look at nginx’s access.log showed a very high level of legitimate-looking traffic, however upon closer inspection I started to notice some patterns.

Identifying Suspicious Requests

The first thing that stood out to me was that many IPs were owned by organizations with simple names like Mosnet LLC, Server LLC, Alax LLC, Express Courier LLC, etc. In addition, I saw many of these IP addresses using different user agents on different requests. I began to look up the IPs in AbuseIPDB.com and with the excellent asn toolkit.

For posterity, here is a list of associated information — such as websites and network names — I could find for each of the organizations owning these IPs:

• Fine Group Servers Solutions LLC (finegroupservers.com)
• TrafficTransitSolution LLC (traffictransitsolution.us)
• Sharktech
• Mosnet LLC (mosnetworks.ru, mosnet.ru)
• ATOMOHOST LLC (atomohost.com, QUALITYNETWORK, Silverstar_Invest_Limited, QuadraNet Enterprises LLC, finegroupservers.com)
• Auction LLC (dauction.ru, blockchainnetworksolutions.co.uk, traffictransitsolution.us)
• LIR LLC (lir.am, traffictransitsolution.us)
• Server LLC (ixserv.ru, finegroupservers.com)
• Alax LLC (alaxona.com, finegroupservers.com)
• Alaxona Internet Inc. (alaxona.com, finegroupservers.com)
• OOO Freenet Group (free.net, vmage.ru)
• Petersburg Internet Network Ltd. (pinspb.ru, depo40.ru, FineTransitRU, QUALITYNETWORK)
• Rustel LLC (rstel.ru, traffictransitsolution.us)
• Express Courier LLC (expcourier.ru, traffictransitsolution.us)
• Sysoev Aleksey Anatolevich (jobbuzzactiv.com, traffictransitsolution.us)
• Bulgakov Alexey Yurievich (UGB, blockchainnetworksolutions.co.uk, finegroupservers.com)
• Dmitry Vorozhtsov (mgn-host.ru, fitz-isp.uk, UGB)
• Nikolaeva Ekaterina Sergeevna (blockchainnetworksolutions.co.uk)
• Atex LLC (atex.ru, blockchainnetworksolutions.co.uk)
• Teleport LLC (teleport.az, blockchainnetworksolutions.co.uk)
• Lovtel LLC (wizardstel.ru, Sharktech, finegroupservers.com)
• Fitz ISP Ltd (fitz-isp.uk, SILVERSTAR-AS)

Eventually a pattern emerged: a majority of these requests were coming from organizations affiliated with either Fine Group Servers Solutions LLC or TrafficTransitSolution LLC. I found others who had written about being on the receiving end of attacks by these organizations:

• Russian DDoS completley unmitigated by Cloudflare
• Идёт атака (May, 2020)

Using a combination of shell script processing and cross checking of the MaxMind GeoIP2 ASN database I found that the IP addresses making these requests were mainly announced by the following ASNs:

• Global Layer B.V. (AS49453)
• ST-BGP (AS46844)
• UGB Hosting OU (AS206485)
• UAB Rakrejus (AS62282)
• AS-COLOCROSSING (AS36352)
• DEDIPATH-LLC (AS35913)
• Silverstar Invest Limited (AS35624)
• ASN-QUADRANET-GLOBAL (AS8100)

Unsurprisingly, most of these organizations and ASNs appear on the Scamalytics list of high-risk ISPs for June, 2021. I have suspicions about several other networks I saw making thousands of legitimate-looking requests in my logs, but these seem to be the major ones.

Mitigating the Attack

In order to block the malicious requests I needed to find some kind of pattern to identify them. Because the requests were using legitimate-looking user agents the only option was to rely on the source IP addresses. Having already identified the ASNs responsible for these IP addresses, I could easily use that to build a perfect block list for this particular attack.

First, I extracted a list of all the unique IP addresses making requests to the server over the past twenty-four hour period (minus some bots known to be OK):

# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 \
| grep -E " (200|499) " \
| grep -v -E "(Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" \
| awk '{print $1}' | sort | uniq > /tmp/all-ips.txt
# wc -l /tmp/all-ips.txt
33830 /tmp/all-ips.txt

Then I used a Python script to look up these 33,830 IP addresses in MaxMind’s GeoIP2 ASN database:

$ ./resolve-addresses-geoip2.py -i /tmp/all-ips.txt -o /tmp/all-ips.csv

Then I used csvgrep and csvcut from the amazing csvkit to extract the networks associated with each IP from any of the matching ASNs:

$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' \
/tmp/all-ips.csv | csvcut -c network \
| sed 1d | sort | uniq > /tmp/networks-to-block.txt
$ wc -l /tmp/networks-to-block.txt
262 /tmp/networks-to-block.txt

I added these 262 networks to a global nginx http block, for example:

...
deny 102.64.120.0/24;
deny 102.64.122.0/23;
deny 107.173.128.0/21;
deny 107.173.32.0/20;
deny 107.174.32.0/19;
deny 107.175.0.0/22;

Note that it would be more efficient to do this at the network layer via iptables (for example), but currently it’s easier for me to deploy changes at the application layer.

After applying these changes to nginx the server returned to a normal load. What is worrying is that these 262 networks represent only a fraction of those announced by the eight ASNs above. According to ipinfo.app there are nearly 4,000 at the time of writing:

$ wget https://asn.ipinfo.app/api/text/nginx/AS49453 \
https://asn.ipinfo.app/api/text/nginx/AS46844 \
https://asn.ipinfo.app/api/text/nginx/AS206485 \
https://asn.ipinfo.app/api/text/nginx/AS62282 \
https://asn.ipinfo.app/api/text/nginx/AS36352 \
https://asn.ipinfo.app/api/text/nginx/AS35913 \
https://asn.ipinfo.app/api/text/nginx/AS35624 \
https://asn.ipinfo.app/api/text/nginx/AS8100
$ cat AS* | sort | uniq | wc -l
3914

I haven’t done the CIDR math, but it’s likely these networks are responsible for millions of IP addresses.

Conclusion

A Twitter user — I won’t link to them — pointed out that this kind of behavior is not illegal in Russia (“and why should it be?”). I don’t understand why people would want to live in a world where it’s seen as acceptable behavior to amass tens of thousands of servers solely for the purpose of brute forcing and nuking others from orbit. We shouldn’t have to hide behind large corporations like Cloudflare, either. The only solution is to get better at defense, I suppose.