server overload: what should I look for?

els · 2008-01-29 21:53:17

A client received a message from his host today, saying that (rough translation) “a server overload had been diagnosed due to a high CPU load and many active scripts and/or website visits (http instances) and/or mySQL queries, caused by a script or application in /httpdocs/textpattern”. Full – relevant – text in Dutch at the bottom of this post (just in case Ruud pops in ;)).

I had a look at the access log and the only strange entries I see are like this:

66.249.66.33 - - [29/Jan/2008:15:22:01 +0100] "GET /klanten/textpattern/setup/function.include HTTP/1.1" 200 1343 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

or

66.249.66.33 - - [29/Jan/2008:13:22:49 +0100] "GET /contact/textpattern/setup/function.main HTTP/1.1" 200 1343 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

There are six of them, from this morning. They vary, but /textpattern/setup/ is always in there. ‘klanten’ and ‘contact’ are sections. Why should googlebot look for non-existing urls?

The client doesn’t remember anything strange happening, except that today he has not been able to upload flv files though the files tab, something that never caused problems before. The upload stops halfway without any notice or error message.
Because the provider came on rather strong (‘don’t do that again, or else…’) we’d like to find the cause asap, but I honestly don’t know what to look for.

Text in Dutch for whoever can read it:

Er is zojuist door een van onze medewerkers overlast geconstateerd van een script of applicatie in /httpdocs/textpattern op uw hostingpakket. Dit zorgde ervoor dat de server waarop uw website geplaatst is, overbelast raakte door o.a. een hoge CPU load en veel draaiende scripts en/of website bezoek (http instanties) en/of MySQL queries.
…
Uw site heeft ervoor gezorgd dat enkele honderden andere sites tijdelijk niet beschikbaar waren. Na het traceren van dit probleem zijn we erachter gekomen dat het in de map “textpattern” zat.
…
De analyse WAAR het op uw site precies mis gaat in die map, is niet iets wat wij kunnen doen; het is namelijk uw site en we gaan er gemakshalve van uit dat u aan de hand van de logfiles zelf kunt zien waar het de mist ingaat.
…
We verzoeken u vriendelijk na te gaan hoe dat dit heeft kunnen ontstaan en de overlast veroorzakende content in deze vorm niet meer online te plaatsen. Wanneer anders blijkt, en uw website opnieuw voor problemen zorgt, zullen wij genoodzaakt zijn onze dienstverlening wederom te onderbreken ten behoeve van onze clienten. Ziet u deze E-Mail dan ook als ‘gele’ kaart, waarbij we u met klem de ernst van de situatie op het hart drukken.

Last edited by els (2008-01-29 21:56:45)

ruud · 2008-01-30 19:10:54

a server overload had been diagnosed due to a high CPU load and many active scripts and/or website visits (http instances) and/or mySQL queries, caused by a script or application in /httpdocs/textpattern

A few questions:

At what time/date did this overload occur precisely?
If you look at website access statistics, do they show an unusual amount of traffic around that time?
Why does the webserver return a 200 status code (OK, file found) instead of a 404 (file not found)?… have you checked if these files actually exist on the webserver?
High diagnostics please :)
If the webhost managed to track it down to the textpattern folder, how come they don’t give information on where the overload occurred: MySQL, many PHP scripts, few PHP scripts eating a lot of memory. Shouldn’t be too hard for them to supply that information, because when diagnosing an overload, that’s one of the first things you check.

The GoogleBot user agent string can be fake.

kees-b · 2008-01-30 21:11:42

ruud wrote:

The GoogleBot user agent string can be fake.

But the IP is a googlebot
66.249.66.33 => crawl-66-249-66-33.googlebot.com

I should indeed ask your host (dutch? which one?) to be more specific about their findings.

kees

els · 2008-01-30 21:13:34

Hi Ruud, thanks! Some answers:

At what time/date did this overload occur precisely?

The provider’s message saying that they ‘just discovered the overload’ was from 28 Jan. 9.45 pm

If you look at website access statistics, do they show an unusual amount of traffic around that time?

No, nothing unusual there. The access log shows a lot of entries from the client himself that afternoon, he has been trying to upload some flv files, which did not work. He might have tried to upload a file that was too big, but when he realized that he tried some smaller ones and did not manage to upload those either.

Why does the webserver return a 200 status code (OK, file found) instead of a 404 (file not found)?… have you checked if these files actually exist on the webserver?

Of course those files don’t exist :) The host temporarily moved the textpattern folder to a private folder, and later moved it back to the public folder, but I don’t suppose they touched the files.

High diagnostics please :)

Textpattern versie: 4.0.5 (r2466)
Laatste update: 2007-08-13 18:29:39/2007-08-13 18:28:21
Document root: /home/vhosts/xxxxxx.nl/httpdocs
$path_to_site: /home/vhosts/xxxxxx.nl/httpdocs
Pad naar textpattern: /home/vhosts/xxxxxx.nl/httpdocs/textpattern
Permanente link modus: section_title
Tijdelijke bestanden map: /home/vhosts/xxxxxx.nl/httpdocs/textpattern/tmp
Site URL: www.xxxxxx.nl
PHP versie: 4.4.7
GD bibliotheek: 2.0 or higher; ondersteunde bestandssoorten: GIF, JPG, PNG.
Lokale server tijd: 2008-01-30 21:52:59
MySQL: 4.1.21-log
Localisatie: nl_NL.UTF-8
Server: Apache/2.0.53 (Fedora)
PHP server API: cgi
RFC 2616 headers: 
Besturingssysteem van de server: Linux 2.6.12-1.1381_FC3smp
Actieve plugins: mem_randimg_mod-0.6c, tcm_is_this-0.6cm, hak_article_image-0.6.1, zem_contact_reborn-4.0.3.19, zem_contact_lang-4.0.3.6m, bas_flvplayer-0.2m, rss_admin_db_manager-4.0.2, asy_wondertag-0.5, zem_nth-0.1, asy_sitemap-0.7m

Pre-flight controle: 
------------------------
Er zijn enkele gewijzigde Textpattern bestanden aanwezig: 
	/include/txp_log.php
------------------------

inhoud .htaccess bestand: 
------------------------
#DirectoryIndex index.php index.html
#Options +FollowSymLinks
#RewriteBase /relative/web/path/

<IfModule mod_rewrite.c>
 RewriteEngine On
 RewriteCond %{REQUEST_FILENAME} -f [OR]
 RewriteCond %{REQUEST_FILENAME} -d
 RewriteRule ^(.+) - [PT,L]

 RewriteRule ^(.*) index.php
</IfModule>

Redirect permanent /home/welkom-bij-xxxxxx http://www.xxxxxx.nl
Redirect permanent /home/ http://www.xxxxxx.nl
Redirect permanent /portfolio/test http://www.xxxxxx.nl/portfolio/topslapen
Redirect permanent /partners/partners http://www.xxxxxx.nl/partners/
Redirect permanent /elevator-pitch/aanbieding http://www.xxxxxx.nl/elevator-pitch/

#php_value register_globals 0

php_value upload_max_filesize 100M
php_value post_max_size 100M

------------------------

Charset (default/config): latin1/utf8
character_set_client: utf8
character_set_connection: utf8
character_set_database: latin1
character_set_results: utf8
character_set_server: latin1
character_set_system: utf8
character_sets_dir: /usr/share/mysql/charsets/
17 Tables: -

PHP uitbreidingen: yp, xml, wddx, tokenizer/0.1, sysvshm, sysvsem, standard/4.4.7, sockets, shmop, session, pspell, posix, overload, ncurses, mysql, mbstring, iconv, gmp, gettext, gd, ftp, exif/1.4 $Id: exif.c,v 1.118.2.37.2.7 2007/01/09 11:38:04 tony2001 Exp $, domxml/20020815, dio/0.1, dbx, dba, curl, ctype, calendar, bz2, bcmath, zlib/1.1, pcre, openssl, ionCube Loader

/include/txp_category.php: r2243 (3706fea923cd77f7053f7803de169df4)
/include/txp_plugin.php: r1917 (c63f72f33986c08367672fc9fe7b42dd)
/include/txp_auth.php: r2356 (33255ec1ea1a825163c78272496d8783)
/include/txp_form.php: r1913 (ecea3fecf9d7d1f8088cda67f097eceb)
/include/txp_section.php: r1891 (1f0121b3e2969d94bc8a7fb98bfdfbd5)
/include/txp_tag.php: r2260 (1bd67bdb9dcfb72e34ea967e39406216)
/include/txp_list.php: r2450 (997a3b1bec7115bf49b76f62b28da146)
/include/txp_page.php: r2099 (56bde34b6c7bcb9123ac91e73065e894)
/include/txp_discuss.php: r2451 (91e0b29ef39a9471ae5c78d0b1bba086)
/include/txp_prefs.php: r2405 (a4b76476930b2376199f23fbfd5f1ac9)
/include/txp_log.php: r2439 (268f2494ce2459b3e2c6fa42e1bd9635)
/include/txp_preview.php: r1238 (696728f35f3557b648c011bb4d6496c3)
/include/txp_image.php: r2439 (9fac6ed0d9d4c3d8196492051f38dc9a)
/include/txp_article.php: r2453 (bdac8fcac5df2f93f10afa7e50c3fb6f)
/include/txp_css.php: r2403 (4e8c52bb1cf5bfe2e2f0640892f9b92e)
/include/txp_admin.php: r2403 (f8700a3d453ece08e7f137b47c967eda)
/include/txp_link.php: r2463 (0a0171bf606296106332d3fdcb83a678)
/include/txp_diag.php: r2361 (dccf3269049dd25e59afdd7ad8d235cd)
/include/txp_file.php: r2403 (e62abd5fcadabe629322ed17135d89eb)
/include/txp_import.php: r1238 (70a6207c0f3604ecfc4b20369986c4d7)
/lib/admin_config.php: r1747 (a2eb09f94d7902a6e95750fc4abcea17)
/lib/txplib_misc.php: r2464 (615afd44a10311f1c0b7852d9bc15d24)
/lib/taglib.php: r1535 (9b519f9dc88791e5ee8eacc029dd6975)
/lib/txplib_head.php: r2404 (2e067b25997cf67cddbdd365570e69d5)
/lib/classTextile.php: r2462 (a031e2ea894e339711c601f230c5ee71)
/lib/txplib_html.php: r2403 (97e173da3058b438513df67fd7d1ceca)
/lib/txplib_db.php: r2406 (5ed67642f805639b54e381fb22efd208)
/lib/IXRClass.php: r765 (137b91497628f0058a2fca9eba5c3b7f)
/lib/txplib_forms.php: r2403 (438a734b52acef40b36d8a3ba23987e8)
/lib/class.thumb.php: r2329 (b2a2fda54371dbd6c40ba553941f090e)
/lib/constants.php: r2361 (ab6d51668fab1e3c98e7d520b1a59f0f)
/lib/txplib_update.php: r1239 (10f28a986d23187b436369dc29ab552f)
/lib/txplib_wrapper.php: r2286 (419125ec74a17a70bf1e86ebfcd45253)
/publish/taghandlers.php: r2444 (cc9de8f2018b01398a2ba542c5f5bdc6)
/publish/atom.php: r2402 (46c4402717f695fde0d49d806adfa4c4)
/publish/log.php: r1637 (5254d0f3942086bc55723923307a51db)
/publish/comment.php: r2460 (2d1ae1dec0784f044e7005fa5ed50930)
/publish/search.php: r1748 (8c86ebcb5be08e214d81ca15a32164ca)
/publish/rss.php: r2393 (09aac29bf22ffa71c1e118e851cff3c3)
/publish.php: r2436 (7087864f1e7c6efe096d3b8e07c350b1)
/index.php: r2466 (30ecf35de5c1edc6ef68e780c8c79daa)
/css.php: r944 (8beba8f83a091068723435cdcdc02f2f)

(I modified php_log.php so it doesn’t log the clients own visits.)

If the webhost managed to track it down to the textpattern folder, how come they don’t give information on where the overload occurred: MySQL, many PHP scripts, few PHP scripts eating a lot of memory. Shouldn’t be too hard for them to supply that information, because when diagnosing an overload, that’s one of the first things you check.

I shall ask the client to ask his host about this.

The GoogleBot user agent string can be fake.

I did an IP check, it looks like Google. But these entries happened afterwards.

Last edited by els (2008-01-30 22:54:31)

ruud · 2008-01-30 21:30:46

He might have tried to upload a file that was too big, but when he realized that he tried some smaller ones and did not manage to upload those either.

Looking at the 100MB upload limit (see diagnostics), I wonder if that caused the overload. I think uploading large files through TXP isn’t very efficient (FTP works better) and downloading very large files through TXP isn’t a really good idea either IMHO.

els · 2008-01-30 22:50:28

ruud wrote:

I think uploading large files through TXP isn’t very efficient (FTP works better) and downloading very large files through TXP isn’t a really good idea either IMHO.

The site is using bas_flv_player to show the videos. So the files must be uploaded through TXP I think. The largest file in there is about 30MB. But suppose he tried to upload a file that was more than 100MB, could that have caused a server overload?

The site also gets quite a lot of hits through Adwords, but I don’t see how that could be related to the problem.

Apart from the googlebot log entries, I can’t find anything unusual. We’ll go back to the host and ask them why they think the trouble came from this account if they can’t even tell what caused it. It’s probably not even TXP related, so thanks for thinking with me despite that :)

kees-b: it’s hosting2go.

Edit: we hope to get some answers from the host tomorrow. I also sent them the strange log entries.
I found two more:
"GET /function.main HTTP/1.1" 200 and "GET /function.include HTTP/1.1" 200, both with the client’s IP… These happened before the googlebot ones.

Last edited by els (2008-01-31 00:12:01)

jm · 2008-01-31 02:12:44

Els wrote:

The site is using bas_flv_player to show the videos. So the files must be uploaded through TXP I think.

Els, your client should be able to FTP the file to the files/ directory, then go to content>files and import the uploaded file.

els · 2008-01-31 08:48:13

Ah, I didn’t think of that, thanks jm!

kees-b · 2008-01-31 09:09:12

ngungo from the joyent forum has set up a very nice and lightweight monitoring service on his accellerator. You can get a free account for 3 sites. You can either monitor your site performance or check server response time by installing a small file to monitor.

els · 2008-01-31 11:05:01

Thanks Kees, I created an account.

els · 2008-01-31 20:14:42

Can someone please explain to me why this is in diagostics:

php_value upload_max_filesize 100M
php_value post_max_size 100M

and phpinfo says:

post_max_size		10M	10M
upload_max_filesize	50M	50M

Also this is probably too low if large files need to be uploaded I think?

max_execution_time	30	30
max_input_time		60	60

Edit: one more question: I have now changed upload_max_filesize to 200M and max_execution_time and max_input_time to 360.
Should I in Advanced preferences change the maximum file size to 200000000 or should that be exactly… er… 209715200 …? Or does it even matter what it says in preferences?

Sorry, one more: diagnostics still say php_value upload_max_filesize 100M, and it’s definitely 200M in phpinfo now.

Last edited by els (2008-01-31 20:30:14)

els · 2008-01-31 22:09:16

OK, with the new settings (max file upload 200MB), trying to upload a file through TXP (flv, 15MB) gave an internal server error. This is in the error log:

[Thu Jan 31 22:02:55 2008] [error] [client 86.88.43.81] Premature end of script headers: index.php, referer: http://www.qubemediaproducties.nl/textpattern/index.php?event=file

Would it help if I upgrade TXP to 4.0.6rc1? Any other ideas?

Textpattern CMS

Textpattern CMS support forum

#1 2008-01-29 21:53:17

server overload: what should I look for?

#2 2008-01-30 19:10:54

Re: server overload: what should I look for?

#3 2008-01-30 21:11:42

Re: server overload: what should I look for?

#4 2008-01-30 21:13:34

Re: server overload: what should I look for?

#5 2008-01-30 21:30:46

Re: server overload: what should I look for?

#6 2008-01-30 22:50:28

Re: server overload: what should I look for?

#7 2008-01-31 02:12:44

Re: server overload: what should I look for?

#8 2008-01-31 08:48:13

Re: server overload: what should I look for?

#9 2008-01-31 09:09:12

Re: server overload: what should I look for?

#10 2008-01-31 11:05:01

Re: server overload: what should I look for?

#11 2008-01-31 20:14:42

Re: server overload: what should I look for?

#12 2008-01-31 22:09:16

Re: server overload: what should I look for?

Board footer