Slicehost Forum - apache and mod_php

apache and mod_php

mandric — Wed, 01 Apr 2009 19:07:06 +0000

I have some re-architecting to do with my current setup. Right now on a 2G slice, I can only run 50 apache processes, and that seems to hit its limit every couple of days where there are 50 processes all serving requests which I can see with /server-status, and service gets highly degraded.

I would like to run a threaded Apache but I do not know if that is stable with mod_php. If it's not then I will likely need to do a nginx proxy, which I'm trying to avoid just because of the extra configuration, etc, needed to maintain it. I'm really trying to avoid nginx since apache has served me greatly and I think this is a mod_php issue that can be handled. Also the apps I'm currently supporting, one of them drupal based, has media files all over the place, so configured another server to intercept media requests would be more of an art than a science.

I've been happy with a streamlined apache but that is near impossible with mod_php and apache prefork. I'm not sure if it's a configuration problem or if it's just a result of the architecture I've chosen. An alternative I've had limited success with php is mod_fcgi, or fastcgi. I've had great success with my django apps using mod_fcgi but php has been unstable. But I haven't run it on a production site yet ... Ideally I would use mod_fcgi for my php apps and mod_wsgi for my python apps. This keeps Apache quite lean and on top of that if it's threaded then it should be able to handle lots of traffic without the memory limitations that apache prefork has. So this is my dilemma, I may need to invest in a new slice to play around with a threaded apache and then migrate stuff. But this is at least a week of work, any advice is appreciated.

apache and mod_php

mandric — Thu, 02 Apr 2009 18:27:08 +0000

well i got more responses on HN than here ... so add your bits there.

http://news.ycombinator.com/item?id=543614

apache and mod_php

blender — Thu, 02 Apr 2009 18:31:51 +0000

Pretty hard to come to any conclusions.

Maybe you're waiting on disk I/O?

What's your iowait under this load?

Cheers

apache and mod_php

mandric — Thu, 02 Apr 2009 18:45:23 +0000

I don't know never used iowait... I'll look into it. What command do I use?

apache and mod_php

blender — Thu, 02 Apr 2009 18:45:33 +0000

Your comment from HN:

"I don't know what the bottleneck is really, that is part of the problem. But I know that every now and then when we we get bursts of traffic I can find with a /server-status that 50 apache processing are in a write state."

I think you do know what the problem is ;-)

What is your PHP Error handling and logging set to? How big are your Apache access_log and error_log files?

Cheers

apache and mod_php

blender — Thu, 02 Apr 2009 18:51:58 +0000

You can use top and look at % "wa" or something like

iostat -x 3

Which will give you lots of disk related info including %iowait

That particular command will give you the stats every 3 seconds until you Ctrl-C out of it.

Cheers

apache and mod_php

mandric — Thu, 02 Apr 2009 19:04:17 +0000

Well all my log files are rotating normally as far as I know ... here's the global log files. I love logrotate.d.

slice:~# ls -al /var/log/apache2/error.log
-rw-r----- 1 root adm 280975 Apr  2 12:05 /var/log/apache2/error.log
slice:~# ls -al /var/log/apache2/access.log
-rw-r----- 1 root adm 1190895 Apr  2 13:50 /var/log/apache2/access.log

The only bottleneck I have seen is in mysql ... which is hard to debug.

http://andric.us/load/localdomain/localhost.localdomain.html

So you think there is definitely something that is hanging up the apache processes. So my next question is,if it happens again, is there a way to poke apache to figure out what is the processes doing besides /server-status which just tells me they are all in W (writing) mode?

apache and mod_php

blender — Thu, 02 Apr 2009 19:25:08 +0000

"So you think there is definitely something that is hanging up the apache processes."

Not necessarily.

The overall impression I get from your munin stats is that your bottleneck is the disk.

You do have iowait - look at the pink in the CPU Usage graphs.

High interrupts for blkif - which is some block device driver for Xen I gather.

One thing you might want to look at is temporary tables created on disk by MySQL - this can really hurt an application.

Run this:

mysqladmin -u root -p ext -ri 30 | grep Created_tmp_disk

First run will tell you how many on-disk temporary tables were created since last restart of MySQL. Then it will tell you how many are created in the 30 seconds time window (until you Control-C out of it).

Cheers

apache and mod_php

mandric — Thu, 02 Apr 2009 19:36:33 +0000

here's the output I let it run for a couple mins

 slice:~# mysqladmin -u root -p ext -ri 30 | grep Created_tmp_disk
 Enter password: 
 | Created_tmp_disk_tables           | 130498      |
 | Created_tmp_disk_tables           | 32          |
 | Created_tmp_disk_tables           | 11          |
 | Created_tmp_disk_tables           | 19          |
 | Created_tmp_disk_tables           | 10          |

apache and mod_php

blender — Thu, 02 Apr 2009 19:50:46 +0000

I would recommend that you run the MySQL tmpdir on a RAM based file system (e.g. tmpfs)

Note there is a current MySQL bug (http://bugs.mysql.com/bug.php?id=30287) that doesn't honor the tmpdir variable - so make sure you are using a version of MySQL that isn't affected.

Look into what I'm suggesting here - I think it will help you a lot.

Cheers

apache and mod_php

mandric — Fri, 03 Apr 2009 21:23:46 +0000

So the problem is mysql. http://andric.us/load/localdomain/localhost.localdomain.html

I get these slow queries every now and then, probably from the insane drupal install, and then apache gets hung up. Either because of IO or db. So I can do a mysqlslowqueries ... here's some of the output.

Count: 1 Time=46.00s (46s) Lock=0.00s (0s) Rows=5.0 (5), windyc[windyc]@local host

SELECT * FROM node n LEFT JOIN radioactivity dn ON n.nid = dn.id LEFT JOIN drigg_node dn2 ON dn2.dnid = n.nid WHERE ( N=N AND !(n.type='S' AND dn2.killed =N ) ) AND ( n.type='S' AND status=N AND promote=N ) ORDER BY dn.energy DESC LIMIT N

I will look into the radioactivity module then on one of the drupal installs i guess.

apache and mod_php

blender — Sat, 04 Apr 2009 14:59:55 +0000

Well, I think it's pretty clear it is I/O as a result of the DB. Look at the load average too - it pretty much mirrors the high interrupts on the blkif device.

The MySQL tmpdir on tmpfs will help but it is treating the symptom - you are right to investigate the queries.

http://dev.mysql.com/doc/refman/5.0/en/internal-temporary-tables.html

We run Drupal too and it is highly relational. Another thing you might consider to speed it up is use the memcache module and get the cache tables out of the DB.

Cheers

Cheers

apache and mod_php

mandric — Sat, 04 Apr 2009 22:40:49 +0000

Yep sounds good. Unfortunately I'm running etch so ...

slice:~# apt-cache policy mysql-server
mysql-server:
  Installed: 5.0.32-7etch1
  Candidate: 5.0.32-7etch8
  Version table:
     5.0.51a-24~bpo40+1 0
          1 http://www.backports.org etch-backports/main Packages
     5.0.32-7etch8 0
        500 http://ftp.us.debian.org etch/main Packages
        500 http://security.debian.org etch/updates/main Packages
 *** 5.0.32-7etch1 0
        100 /var/lib/dpkg/status

I may have to go with a backport or move off etch to get the mysql tmpfs patch you mentioned.

apache and mod_php

alangiv — Sun, 05 Apr 2009 14:54:24 +0000

You can get MySQL 5.1.32 from http://www.dotdeb.org/ Just add the appropriate repository to Apt. I installed it on Ubuntu 8.04 without any problems. MySQL, Apache, etc not being updated via the regular repositories sure is a real pain.

apache and mod_php

mandric — Sun, 05 Apr 2009 20:10:54 +0000

Are there any issues with upgrading from mysql 5.0 to 5.1? Just a minor rev right, my dbs won't break?

apache and mod_php

alangiv — Mon, 06 Apr 2009 09:34:41 +0000

I had no issue upgrading from 5.0 to 5.1. I backed up all user databases and the system databases just for safety, also expecting that I would need to restore them afterward, however they were upgraded in-place.

After configuring the correct repositories from dotdeb.org, I ran "apt-get -y install mysql-server-5.1" and had no problems at all. Not bad considering other people have had issues with upgrading Ubuntu specifically - the repositories were designed for Etch and Lenny.

apache and mod_php

blender — Mon, 06 Apr 2009 15:53:56 +0000

According to the MySQL bug it "has been introduced in version 5.0.44"

So you should be okay with your current MySQL.

Cheers

apache and mod_php

mandric — Mon, 06 Apr 2009 17:10:19 +0000

Hah! Sorry for the noise ... ok back to tmpfs and memcache then!

apache and mod_php

mandric — Mon, 06 Apr 2009 17:44:20 +0000

blender, so you mentioned "Another thing you might consider to speed it up is use the memcache module and get the cache tables out of the DB", I just want to clarify what you meant by this. Am I right in thinking that I can configure mysql to use memcached as its tmpfs rather than disk? what did you mean by "memcache module"? I assumed http://danga.com/memcached/ .

apache and mod_php

blender — Tue, 07 Apr 2009 19:35:33 +0000

Sorry, by memcache module I meant the Drupal memcache module:

http://drupal.org/project/memcache

You do need to run memcached so that the Drupal module can use it for the cache tables (i.e. cache, cache_block, cache_filter, etc.) instead of MySQL.

Cheers