Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#1 2020-03-26 08:28:59

gaekwad
Admin
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 2,786

Textpattern web server under increased load

Please be aware there may be some slowness associated with accessing Textpattern websites for the near future. There appears to be a fleet of bots hammering us with garbage GET and PUT requests.

Our web server is configured to deny all PUT requests, so they’re not causing problems, but the GET requests are. Log files are increasing at about 3-5KB/s – not massive, but much more than I’m used to seeing.

I’m working on a fix. Thanks for your patience.

Update #1: I think it’s fixed. Server load back to normal. Please report any problems below.

Last edited by gaekwad (2020-03-26 08:48:00)

Offline

#2 2020-03-26 09:27:13

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 7,776
Website

Re: Textpattern web server under increased load

Hi Pete, would this be of help?


Yiannis
——————————
neme.org | hblack.net | LABS | State Machines | NeMe @ github | Covid-19; a resource
I do my best editing after I click on the submit button.

Offline

#3 2020-03-26 12:34:13

gaekwad
Admin
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 2,786

Re: Textpattern web server under increased load

colak wrote #322299:

would this be of help?

Thanks, Yiannis- we run on Nginx, not Apache. There’s already some rate-limiting code in place to reduce the impact of bots generally. This one seems to be limited to one IP address on a GoDaddy VPS rack. I’ve blocked it, reported it to their abuse team, so not much more I can do beyond monitoring the graphs etc for anomalies and responding to alerts.

Most of the incidents I deal with on our server are short term, usually opportunists looking for credentials in dot files. Take this morning’s contestant, for example (excerpt):

2020/03/26 08:16:32 [error] 5932#5932: *50769 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /apitude/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50770 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /b2b/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50773 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /biblicaltours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50774 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /brochures/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50775 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /blog/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50777 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /cappadociatours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50771 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /athenstours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50772 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /biblical_asia_minor/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50768 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /acentedokuman/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50776 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /cgi-bin/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:49 [error] 5932#5932: *51712 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /ferrytosamos/.env HTTP/1.1", host: "forum.textpattern.com"

The relevant part is the URL stub of the GET request: they’re all hidden .env files, typically used for storing API keys. There are thousands of requests for these every day, all of which are denied by the web server, so usually it’s all handled without fanfare. But today, the combination of the dot files scan plus many thousands of GET requests that resulted in a 404 error gave a CPU spike to 100% for about 15 minutes, then off-and-on 100% for another 15 minutes or so.

Looking at the logs, each of our sites was hit in turn for a few minutes. No evidence of any breach or compromise, just a lot of log file entries over about 30 minutes, then back to normal.

Offline

Board footer

Powered by FluxBB