Approaches against DDoS attacks?

jakob · Today 12:03:57

I inherited a client a while back that last week was, and again currently is the target of a DDoS attack. There’s nothing about the site that is at all contentious and the owner runs a small business that I don’t think has a market share that anyone would put the effort into destroying so the reason is probably just bad luck.

Looking at the log files from yesterday, there were around 9 million requests at millisecond intervals that always use an url made of successive bits of other urls on the same site, e.g.:

/section/123/article-name/section/456/other-article-name/section/789/and-so-on

or using url parameters that we don’t use like ?lang=more-url-splurge.

At that point the host takes the site offline, and I’ve crafted various regex patterns to sink non-bona-fide url requests before textpattern tries to process them, but the host says it happens again (though I’ve not been able to corroborate that because the site was taken offline again).

The host has suggested we pass the domain through Cloudflare. This is not a hobby site, but also not a huge seller, e.g. it shows wares for a business but doesn’t have an online shop on the homepage itself, so I’m unsure whether Cloudflare’s free plan applies here. I guess not. That adds 20$ a month to the client’s outgoings. The site is currently hosted on a shared hosting plan.

Do I have any other sensible options? Could, for example, relocating to another host help here, or do I just set end up getting targeted on the new host? Are there hosts that do both cloudflare’s ddos filtering and the hosting part for an affordable fee?

Any advice gratefully received.

gaekwad · Today 13:33:57

jakob wrote #342845:

The host has suggested we pass the domain through Cloudflare. This is not a hobby site, but also not a huge seller, e.g. it shows wares for a business but doesn’t have an online shop on the homepage itself, so I’m unsure whether Cloudflare’s free plan applies here. I guess not.

Priority 1 is business continuity. Try this:

Sign up customer with free Cloudflare account.
Make backup / offline copy of all DNS records and name server entries.
Add domain to Cloudflare, import existing DNS records.
Flip nameservers to Cloudflare.
Check the web logs for changes in the amount of garbage.
Confirm with host that you have followed their advice and site now routes through Cloudflare.

Try the free CF plan. If there’s no change, you can trivially flip the name servers back to their previous home and you’ve not lost any $ in service charges.

Is the host a well-known / big organisation? If they’re recommending CF then it follows that they’re not recommending a web application firewall on their own infrastructure…they want CF to do the heavy lifting. Web server rules are still hitting the web server, and at millisecond intervals on shared hosting that’s going to sting.

Is the attack hitting an IP address, the (edit: shared server) hostname, or the client domain? If either of the first two, maybe time to lift & shift to another host. If it’s the domain, that won’t help.

Last edited by gaekwad (Today 13:41:08)

gaekwad · Today 13:39:21

jakob wrote #342845:

Do I have any other sensible options?

I haven’t used it myself, but KeyCDN has a reportedly competent (D)DoS service, and they’re Swiss-based.

jakob · Today 13:40:59

Thanks. The GET requests are all to the domain.

I had a look at cloudflare alternatives (especially european) and all the one’s I could find that even quote prices were an order of magnitude more expensive.

I saw that OVHcloud and Hetzner say they have ddos protection free as part of their plans. Is something like that at all realistic?

gaekwad · Today 13:43:41

jakob wrote #342848:

I saw that OVHcloud and Hetzner say they have ddos protection free as part of their plans. Is something like that at all realistic?

Tread carefully with Hetzner. They’re very competent and I actively use them, but they have a history of closing accounts with little to notice where there’re breaches of terms and conditions – especially newer accounts. If you approach them formally with a background to what you have going on, they will more than likely err towards co-operation – dialogue with a pre-sales person will set expectations for all concerned.

jakob · Today 13:45:03

So you think I could use cloudflare free first, although ostensibly a small business, then upgrade if relevant. There’s been no history of attacks previously in the past 3-4 years I’ve been involved with the site.

gaekwad · Today 13:47:19

jakob wrote #342850:

So you think I could use cloudflare free first, although ostensibly a small business, then upgrade if relevant.

Yes. I have multiple business clients on the free CF tier, never had a problem.

jakob · Today 13:56:45

Thanks. I’ll try that then. Presumably you sign them up as individual customers?

KeyCDN say they offer DDOS protection but I find nothing concrete, no tutorials etc. except for information pages.

gaekwad · Today 14:07:13

jakob wrote #342852:

Presumably you sign them up as individual customers?

Yes. Here’s what I’d do:

Sign up at CF as you with your own (agency) domain.
Sign client up at CF with their own (business) domain.
Log in as client, add yourself as an admin collaborator. Log out.
Log in as you, accept collab invite, and you’ll see their domain(s) in their own silo as an account.

This sidesteps any weirdness if they go with someone else, and they retain the keys to the kingdom.

Here’s what my CF accounts page looks like, each line is a separate collaborator project / client (and my own is named accordingly):

Last edited by gaekwad (Today 14:28:04)

jakob · Today 14:53:11

Brilliant. Good tips. Thank you! Turns out the host that recommended fronting the site with cloudflare doesn’t permit you to change their nameservers. So, I’ve had to call support and I’m waiting in their helpline loop … … … …

gaekwad · Today 14:56:28

jakob wrote #342854:

Turns out the host that recommended fronting the site with cloudflare doesn’t permit you to change their nameservers.

Yikes. Consider splitting domain registration and hosting vendors in the future. You lose a little of the convenience, but it’s no longer a single point of failure. Bonus points if you carve off email to a proper provider, too. Three vendors, so a bit more work, but much more flexibility.

Also…shared hosting that doesn’t let you change name servers to outbound…that’s a good sign that moving the client to other hosting would be sensible in the future.

jakob · Today 20:39:01

Thanks for your speedy help today.

The site is now back online fronted by Cloudflare, but it’s seemingly not over yet. In the 1-2 hrs it’s been back online there have already been 135k page requests, mostly from Singapore. Currently it’s on the “I’m under attack” mode until I work out how to add some better rules to Cloudflare to stem the flow.

In other news, I’ve learned that owner has other plans for his life, so researching new hosting options don’t make sense. That’s the only only client I had on that host. They were at least helpful today, even if I had to wait 25 minutes to get through…

gaekwad · Today 20:43:06

jakob wrote #342856:

even if I had to wait 25 minutes to get through…

…they might have been busy learning how to change name servers…

jakob · Today 22:31:01

gaekwad wrote #342857:

…they might have been busy learning how to change name servers…

Haha.

It seems most of the traffic is coming from Singapore and from bytedance (tiktok) servers. 95% is blocked, but as there have been 2.5 million requests in the last 3hrs, some must still be getting through, so the Cloudflare “I’m a person”-captcha is still on. I wanted to add a ratelimit rule but I’ve not worked out properly how to do it, or maybe it doesn’t help so much when the ip is constantly changing.

My attempt to use a regex to restrict to bona-fide url requests was not allowed … that’s only possible with a paid account they say.

Textpattern CMS

Textpattern CMS support forum

#1 Today 12:03:57