Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Textpattern web server under increased load
Please be aware there may be some slowness associated with accessing Textpattern websites for the near future. There appears to be a fleet of bots hammering us with garbage GET and PUT requests.
Our web server is configured to deny all PUT requests, so they’re not causing problems, but the GET requests are. Log files are increasing at about 3-5KB/s – not massive, but much more than I’m used to seeing.
I’m working on a fix. Thanks for your patience.
Update #1: I think it’s fixed. Server load back to normal. Please report any problems below.
Last edited by gaekwad (2020-03-26 08:48:00)
Offline
Re: Textpattern web server under increased load
Hi Pete, would this be of help?
Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.
Offline
Re: Textpattern web server under increased load
colak wrote #322299:
would this be of help?
Thanks, Yiannis- we run on Nginx, not Apache. There’s already some rate-limiting code in place to reduce the impact of bots generally. This one seems to be limited to one IP address on a GoDaddy VPS rack. I’ve blocked it, reported it to their abuse team, so not much more I can do beyond monitoring the graphs etc for anomalies and responding to alerts.
Most of the incidents I deal with on our server are short term, usually opportunists looking for credentials in dot files. Take this morning’s contestant, for example (excerpt):
2020/03/26 08:16:32 [error] 5932#5932: *50769 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /apitude/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50770 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /b2b/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50773 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /biblicaltours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50774 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /brochures/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50775 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /blog/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50777 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /cappadociatours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50771 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /athenstours/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50772 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /biblical_asia_minor/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5932#5932: *50768 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /acentedokuman/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:32 [error] 5933#5933: *50776 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /cgi-bin/.env HTTP/1.1", host: "forum.textpattern.com"
2020/03/26 08:16:49 [error] 5932#5932: *51712 access forbidden by rule, client: [SNIPPED], server: forum.textpattern.com, request: "GET /ferrytosamos/.env HTTP/1.1", host: "forum.textpattern.com"The relevant part is the URL stub of the GET request: they’re all hidden .env files, typically used for storing API keys. There are thousands of requests for these every day, all of which are denied by the web server, so usually it’s all handled without fanfare. But today, the combination of the dot files scan plus many thousands of GET requests that resulted in a 404 error gave a CPU spike to 100% for about 15 minutes, then off-and-on 100% for another 15 minutes or so.
Looking at the logs, each of our sites was hit in turn for a few minutes. No evidence of any breach or compromise, just a lot of log file entries over about 30 minutes, then back to normal.
Offline