smd_prognostics: monitor your Txp installation for suspicious activity

Bloke · 2010-11-11 02:00:56

You know that horrible shivery feeling you get when someone’s been at your files? Wouldn’t it be nice to be informed that something dodgy was going on so you could get on top of it right away instead of only finding out when Google slapped a “This site may harm your computer” warning on your links?

In steps smd_prognostics: pro-active diagnostics for Textpattern. It requires TXP 4.4.1+ and PHP 5.

This beastie monitors your site’s files and sends out an alarm when things change. You can acknowledge the alarms and the plugin will then go about its business until the next time something changes. Of course, you can configure how frequently you’re nagged and whether you want to send off the forensic prognostics (a.k.a. frognostics) to yourself, or me to help improve the plugin.

I should warn you up-front that the type of stuff the frognostics sends may be quite sensitive (file paths and stuff) so if you don’t want me to see that sort of thing, don’t put my prognostics e-mail address in! You can always sanitize the data and send it by hand later if you prefer, or cut me out of the loop completely. The dedicated e-mail account I set up is only there so you can help me improve the plugin — and perhaps Txp — by sending intrusion detections for me to analyse.

The plugin also has a few real-time monitors that try to detect common attacks and block them. I would love to hear your reports on whether it works or it’s rubbish. There is also an advice page on how to harden your installation (more in later versions as I get my head round it), and even a simplistic but entertaining password strength monitor on the Admin->Users tab.

smd_prognostics is not the ultimate tool for peace of mind and you should never let your guard down. But it can be useful to help you recover quickly should the unthinkable happen as it allows you to very rapidly find the changes and fix them. Just ask kevinpotts who was my gracious test subject how useful it was when it detected a rather nasty variant of the c99shell backdoor on his server (probably let in by someone else on the same shared host).

Of course I couldn’t have done this alone. I am also indebted to Steve (net-carver) for his unending support, ideas and enthusiasm in helping me take this plugin far beyond the initial capabilities I sent to him in the alpha version. He’s a true hero.

With all that said, download, install, and read the help file to get the most out of the plugin; I’ve tried to keep things brief, honest! Above all, please leave feedback here so I can shape this thing in future versions. If anyone has any guidance on what the plugin can look for, any extra advice it can give (for example, Steve has plans for helping me check MySQL’s SHOW GRANTS capabilities in a meaningful way) or any other info on how to improve the plugin workflow then let me know.

It’s taken the best part of six weeks to hammer this into shape between my own testing and those of my willing beta testers; hope it’s useful :-)

Revision history
————————

All available versions and changes are listed here. Each entry indexes the relevant post(s) in the thread to learn about the features.

11 Nov 2010 | 0.10 | Initial release
11 Nov 2010 | 0.11 | Added Ignore button
12 Nov 2010 | 0.12 | Fixed white screen of death on Files Save (binary files are now left unprocessed) ; improved performance ; added file quantity check
15 Nov 2010 | 0.13 | Alarms panel now always displays all alerts and doesn’t interfere with file checking rotation ; fixed incorrect URL in acknowledge messages (thanks thebombsite) though multi-site installations may still be wrong ; removed dumb admin-side SQL injection, added Check files between and TXP version advice (all thanks ruud) ; tweaked injection detector ; refactored e-mail header code
16 Nov 2010 | 0.14 | Added forensic options for header spoofing/SQL injections ; can now throw HTTP response code or custom message/form instead of ‘nice try’ ; wildcard ability for ignored files ; added user check and TXP dir option (thanks maverick)
09 Dec 2010 | 0.15 | Improved warning display when saving preferences (thanks maniqui) ; skipped comment preview step for SQL injections ; fixed version number in frognostics ; fixed display error on Advice panel ; added sql_injection callback ; added rpc advice check
09 Dec 2010 | 0.16 | Never assume English button names (thanks roelof)
30 Dec 2010 | 0.17 | Added injection sensitivity to try and minimise false positives on sites that accept comments ; fixed loss of settings on switching prognostics subtab
26 Jan 2012 | 0.20 | Performance boost ; added separate XSS shield pref ; altered callback signature: event=“smd_frognostics” to avoid clashes with the admin side ; fixed a few warnings ; password strength meter integrated with smd_user_manager ; fixed array_merge()-requires-array-argument snafu ; added CSRF tokens ; fixed rogue status msg when viewing alarms panel

Last edited by Bloke (2012-01-26 01:42:38)

Bloke · 2010-11-11 14:33:55

New version 0.11 adds the ability to Ignore files when acknowledging alarms. Thus the alarm will be acknowledged but will not automatically add the file(s) to the list of monitored files. Saves you having to acknowledge everything and then visit the Files list to unselect the ones you don’t want to monitor.

jstubbs · 2010-11-11 14:51:31

Wow. Incredible work Stef! But that’s becoming par for the course…

mrdale · 2010-11-11 15:33:10

I hate to be a prognosticator. But something is terribly wrong ;) I get a white screen at http://stefdawson.com

and BTW, zounds man!

Bloke · 2010-11-11 19:21:54

Thanks Jonathan.

@mrdale: freaky. I got it too in Firefox then I checked in Opera and it was fine, went back to Firefox and all was well. Think my hoster must be having problems but I’ll keep an eye on things, thanks for the report.

Last edited by Bloke (2010-11-11 19:24:48)

Bloke · 2010-11-11 19:56:33

Seems my site’s been issuing more than its fair share of 500 Internal Server Errors since lunchtime. Hoster has no mention of it but there’s no common thread I can see in the log files. Different types of request from different hosts and referrers (and bots) hitting different pages all got about 90% 500s, and the odd 200 or 301/302 response in between. Then everything went ok for a spell and started going all 500y again about 3:30pm my time… until just now.

If it was prognostics and it was working properly you’d see a ‘Nice Try’ message for any dubious access attempts. If it’s prognostics and it’s bailing out for some reason then I’ll need to trace it through. If it’s something outside my control then either someone else on my shared server has problems, my host are keeping quiet, or I’ll need to dig further. Gonna keep an eye on the logs tonight and see if I can catch a whiff of anything going on. Apologies for the strange behaviour.

Bloke · 2010-11-11 20:22:30

OK, further investigation reveals that for some reason the public-side click check in prognostics is doing something strange. I set the time out to 60 seconds and public clicks on. I refreshed the page repeatedly and 60 seconds later I got a white screen of death. I continued to get this white screen until I hit the admin side (any tab except Plugins) thus the prognostics routine ran, did something that ‘unlocked’ itself and worked fine.

I can merrily click away on the admin side forever and a day without issue, but if I hit the site from the public side, the first time it triggers the prognostics routine: BAM it dies and locks any further requests until the admin side is visited. Well, any further requests from the same host as it seems that other things can sometimes get through.

Gotta be something odd with the callback (pretext) that’s killing things and the way it interacts with my site. But why it would lock everything out continuously until the admin side unlocks it is a mystery. Also a mystery is why it doesn’t exhibit this behaviour on my dev site which is on the same server running the same versions of everything….

If anyone else experiences anything similar, please let me know as much detail as you can so I might be able to nail this. I apologise in advance if the plugin does break, but the temporary solution is to turn off the Public-side click checking.

I’ll put my code diving gear on and take the plunge later tonight.

Last edited by Bloke (2010-11-11 20:25:13)

maniar · 2010-11-11 20:30:57

I just installed it – everything seems to be working alright. (I’ve the public side clicks turned off)

Got my first message too – Prognostics Checksums.txt was missing – so I will assume it works.

I won’t send @Bloke any forensics data yet – I’d rather see them for myself before sending it to him.

Great work otherwise. I was hoping and expecting for such a feature. Thank you !

kevinpotts · 2010-11-11 22:15:16

I can tell you that I’ve had this installed on my main site for several weeks now, and it has already caught two very nasty changes to textpattern’s core PHP files that would have resulted in ungodly pharma spam. Invaluable. If your host’s security is suspect (wink wink Dreamhost), install this tool immediately.

hcgtv · 2010-11-11 23:12:07

Stef,

I installed it at PHPXref.com. It’s just checking the Textpattern core files, not the cross references.

I have Check files on public side clicks: set to yes and things are working fine so far. This site gets a boat load of attacks, it should be fun to see what it detects. I have check files and alarms set to 3600 seconds for now, just to test. Notify me via email is set, I can add you on it, just let me know the address.

I’ve noticed on the Setup page, that when I make a change, and click save, that it returns the screen with the old values, I change it again, click save a second time, and it saves the correct values.

Kevin,

I’m on DreamHost, let the games begin.

Bloke · 2010-11-11 23:32:17

hcgtv wrote:

I have Check files on public side clicks: set to yes and things are working fine so far.

OK, good to know, thanks. I think the problems on my site may be down to a few factors:

1) some pretty intense .htaccess redirects
2) checking a lot of files (around 700 of them). So perhaps the script is hitting some PHP resource limit or something on the public side due to the amount of other stuff going on (guessing the load is lower on the admin side)

I’m checking things as I go. I’ve also found a wee bug if you happen to try to monitor (large) binary files; the sanitization I do to cater for Windows/UNIX line endings and SVN differences is triggering the white screen of death when you try and Save the Files list. Fix in progress: the workaround is to only include text files for now, or fairly small binary files (not 10Mb video files like I tried!)

I’ve noticed on the Setup page, that when I make a change, and click save, that it returns the screen with the old values

Shouldn’t happen on TXP 4.3.0 which is the plugin’s minimum requirement. That behaviour you’re seeing is a 4.2.0 issue. It is actually saving the values, it just displays them incorrectly the first time round after a Save. Verify this by just clicking the Prognostics tab heading again after saving.

However, if it’s happening under 4.3.0 then please let me know which PHP/MySQL version you’re running, etc and I’ll see if it’s something I can fix.

Regarding the e-mail address to send the frognostics to, it’s in the plugin help towards the end. If you don’t mind the file paths and things divulged then by all means add that address. If not, sanitize the e-mails you get and forward them whenever you get a chance. Thanks for helping out. As you say, will be interesting to see what gets through.

hcgtv · 2010-11-11 23:42:36

Bloke wrote:

Shouldn’t happen on TXP 4.3.0 which is the plugin’s minimum requirement. That behaviour you’re seeing is a 4.2.0 issue. It is actually saving the values, it just displays them incorrectly the first time round after a Save.

Yes, the site is running 4.2.0, haven’t had the chance to upgrade.

Regarding the e-mail address to send the frognostics to, it’s in the plugin help towards the end.

Added.

maverick · 2010-11-12 00:26:37

Stef

I installed it on one of my mulit-sites. When I go to the file panel, it appears I now have access to the files in all of the sites that are in the sites folder. Is this intended behavior?

Thanks

Mike

maverick · 2010-11-12 00:55:27

Another issue that probably relates to being multi-site – the link in the alarm emailed to me doesn’t point back to the admin panel. (admin.domain.com)

Instead it was:

a href=“http://www.domain.com//index.php?event=smd_prognostics&step=smd_prognostics_ack&smd_prognostics_suppress=1”>Acknowledge alarms

Last edited by maverick (2010-11-12 00:55:47)

Bloke · 2010-11-12 00:58:28

maverick wrote:

When I go to the file panel, it appears I now have access to the files in all of the sites that are in the sites folder. Is this intended behavior?

Short answer: I don’t know as I’ve never tried multi-site :-)

Long answer: it just does a recursive listing of every file below the given path(s) in File locations, so check the path there. If there’s no trailing slash it could be that it’s picking up everything at that level and below *shrug*. If it’s causing problems in multi-site you can try listing each site (comma-separated) in the File locations field. e.g. /path/to/site1/, /path/to/site2/, .... That might cut down your files list a bit. Or put the plugin on each site and monitor them all separately (which might be preferred in terms of speed), pooling the checksums files in a central dir just outside the docroot of your core install. Use the Unique prefix option for this so your checksums files don’t clash.

I’m working on speeding the plugin up a bit. Each time the timeout period has been met and the plugin determines its time to check the files, it reads the entire contents of each file you are monitoring, calculates its checksum and compares that against the stored checksum. If you have a lot of files or some large files in your monitor list, it adds to the processing time and slows your site down (admin-side and/or public side depending on the settings). For this reason, only monitor what you really need to monitor and ignore stuff you can live without or that you don’t care about too much.

I’ve not quite got my head round how this plugin should operate in a multi-site environment. Multiple databases, multiple content paths (files, images, etc) but only one set of core files, right? Do you still log into each site separately? Guess you must do. So do you install this plugin on each site separately? I guess you should do. From my (possibly misguided) five-minute think over this, I reckon the best way to run it might be one smd_prognostics per site to cover that site’s files (images, files, whatever) and then nominate one of the sites to also monitor the core files. No need to monitor the core files from all of them.

Any of that make sense? If you have any thoughts on how the plugin coiuld be improved in multi-site environments then please let me know. I’ll see what I can do to simplify things.

EDIT: yah nuts. Yeah it uses hu to return the path to the sitefor acknowledging alarms which I believe is wrong in multi-site. Hmmm. Needs some thought.

Last edited by Bloke (2010-11-12 01:00:01)

Textpattern CMS

Textpattern CMS support forum

#1 2010-11-11 02:00:56

smd_prognostics: monitor your Txp installation for suspicious activity

#2 2010-11-11 14:33:55

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#3 2010-11-11 14:51:31

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#4 2010-11-11 15:33:10

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#5 2010-11-11 19:21:54

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#6 2010-11-11 19:56:33

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#7 2010-11-11 20:22:30

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#8 2010-11-11 20:30:57

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#9 2010-11-11 22:15:16

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#10 2010-11-11 23:12:07

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#11 2010-11-11 23:32:17

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#12 2010-11-11 23:42:36

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#13 2010-11-12 00:26:37

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#14 2010-11-12 00:55:27

Re: smd_prognostics: monitor your Txp installation for suspicious activity

#15 2010-11-12 00:58:28

Re: smd_prognostics: monitor your Txp installation for suspicious activity

Board footer