Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Automatic Referral spam Blocking.
Provided I have time (both work and school permitting) I’ve decided to come up with a means to catch referral spammers and auto-block them. My page has been plagued with referral spam for quite some time, and despite it being just a shell of a site, I’m averaging about two gigs of traffic a month from referral spammers. (and I don’t have any public ally accessible referrer logs!)
Look at the following referral spam from just a few moments ago.
66.55.149.35.choopa.net – - [21/Jan/2005:13:04:09 -0500] “GET / HTTP/1.1” 200 1522 “http://las-vegas-hotel.go.ro/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)”
66.55.149.35.choopa.net – - [21/Jan/2005:13:06:23 -0500] “GET / HTTP/1.1” 200 1522 “http://las-vegas-hotel.go.ro/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)”
This is from a log of over 300 hits within a couple hours time. This seems easy to trap as it is the same IP coming from the same “referrer” numerous times over the course of a period of time. MUCH too often to be any legitimate user accessing my page. I figure a nightly (hourly) parse of the logs would catch this and apply the appropriate ban to the htaccess file. Perhaps even a database table that contains the most current traffic and applies IP banning based upon patterns.
I’m also considering something along the lines of a Bayesian filter (maybe using the Bayesian class from: http://www.phpgeek.com/pragmacms/index.php?layout=main&cslot_1=14) that would trap common terms and apply a 403 error page before processing their request and then placing the appropriate .htaccess ban.
Currently I’m looking for any ideas, suggestions, etc before I start any code. Also, I don’t have much time to devote to this specifically myself, so if anyone else would be willing to work on such a project, let me know.
Offline
Re: Automatic Referral spam Blocking.
This could be helpful in some cases, but the prime referral spammer I see in my logs now is using some form of rotating proxy. Each request comes from a different IP, yet all point to the same invalid referral page.
This particular bozo has dozens if not hundreds of domains (meaning, it’s hard to block them all simply by referring URL), and uses rotating proxies, so you can’t block the issuing IP. Lots more here.
Last edited by reid (2005-01-21 19:15:55)
TextPattern user since 04/04/04
Offline
Re: Automatic Referral spam Blocking.
I get the same thing. I think they’re counting on people having public referral logs, to boost pagerank.
The strange thing about all of them is that they all return something like “Account Terminated”, or something of the like.
The following is true
The above statement is false.
Offline
#4 2005-01-21 19:33:13
- Andrew
- Plugin Author
- Registered: 2004-02-23
- Posts: 730
Re: Automatic Referral spam Blocking.
You should talk to zem (Alex). he is deep in the process of developing an anti-spam tool for Txp.
Offline
Re: Automatic Referral spam Blocking.
I may be dumb, but can someone explain to me what’s the big deal about spam referral ?
Offline
Re: Automatic Referral spam Blocking.
> reid wrote:
> This could be helpful in some cases, but the prime referral spammer I see in my logs now is using some form of rotating proxy. Each request comes from a different IP, yet all point to the same invalid referral page.
I’m thinking of something that not only blocks IP, but also anything coming from referrer spam-domain.com.
> Jeremie wrote:
> I may be dumb, but can someone explain to me what’s the big deal about spam referral ?
Traffic. I’m on a 10 gig account and am already receiving an average of 2 gigs of traffic a month from them alone. If I go over, I start paying additional hosting fees for their garbage.
Offline
#7 2005-01-21 19:48:31
- Andrew
- Plugin Author
- Registered: 2004-02-23
- Posts: 730
Re: Automatic Referral spam Blocking.
Big deal in what way? if you’re asking why people care to do it, that’s easy — and related to the rel="nofollow" — the more links to you, higher rankings. Many people have their refer logs in public view, so it allows them free linkage.
If you’re asking why we care to block, refer spam is just one step removed from comment spam. They’re usually the same people, or at least related. They are the scum of the earth. Do you want links to incest, rape, torture, and bestiality showing up on your site? I sure don’t. Nor do I want them in my refer logs.Last edited by compooter (2005-01-21 19:54:14)
Offline
Re: Automatic Referral spam Blocking.
> PeterS wrote:
> I think they’re counting on people having public referral logs, to boost pagerank.
It’s my opinion this is being done by multiuse spamware. It first tries to leave automated comment spam, and if it fails (as it does at Textpattern sites), it unleashes the Referral Bomb.
> The strange thing about all of them is that they all return something like “Account Terminated”, or something of the like.
That’s a false “admin” page the spammer puts up at first (did you notice the truly odd language? Google search due to miss-proper use to get a rough count on the number of sites this guy is using). They hope you believe the account has been terminated and ignore it. Then in a few days, it changes to a redirect to a drugs/poker/porn site.
TextPattern user since 04/04/04
Offline
Re: Automatic Referral spam Blocking.
Ok for the bandwith usage, that make sense.
But public referral log ?? That’s putting a short skirt, taking off your underwear and bend over for spammers (and everyone, that’s pretty private data… if your website has any good ranking on some common words, that’s even very valuable data).
Offline
Re: Automatic Referral spam Blocking.
This is why it’s now pretty rare to find any public referral log nowadays. Yet this idiots still spout this rubbish.
I seriously wonder what these people are like, do they have any sense of right or wrong? Would one be able to notice a difference between spammers and regular people in real life?
Offline
Re: Automatic Referral spam Blocking.
Public referral logs have gotten rarer, but that hasn’t slowed the onslaught.
Regardless, these bozos make private referral pages (both in Textpattern and in any stats package like Urchin) nearly totally useless. Last Sunday when I looked in Urchin, my top 20 referrers were all spam sites, except one. Google.
TextPattern user since 04/04/04
Offline
Re: Automatic Referral spam Blocking.
Thanks for the roundup of this on your blog, reid. I’ve had an absolutely enormous amount of this referer spam in the past few weeks, too.
-Alan
Offline