apache and mod_php

mandric

I have some re-architecting to do with my current setup. Right now on a 2G slice, I can only run 50 apache processes, and that seems to hit its limit every couple of days where there are 50 processes all serving requests which I can see with /server-status, and service gets highly degraded.

I would like to run a threaded Apache but I do not know if that is stable with mod_php. If it's not then I will likely need to do a nginx proxy, which I'm trying to avoid just because of the extra configuration, etc, needed to maintain it. I'm really trying to avoid nginx since apache has served me greatly and I think this is a mod_php issue that can be handled. Also the apps I'm currently supporting, one of them drupal based, has media files all over the place, so configured another server to intercept media requests would be more of an art than a science.

I've been happy with a streamlined apache but that is near impossible with mod_php and apache prefork. I'm not sure if it's a configuration problem or if it's just a result of the architecture I've chosen. An alternative I've had limited success with php is mod_fcgi, or fastcgi. I've had great success with my django apps using mod_fcgi but php has been unstable. But I haven't run it on a production site yet ... Ideally I would use mod_fcgi for my php apps and mod_wsgi for my python apps. This keeps Apache quite lean and on top of that if it's threaded then it should be able to handle lots of traffic without the memory limitations that apache prefork has. So this is my dilemma, I may need to invest in a new slice to play around with a threaded apache and then migrate stuff. But this is at least a week of work, any advice is appreciated.

mandric

well i got more responses on HN than here ... so add your bits there.

http://news.ycombinator.com/item?id=543614

blender

Pretty hard to come to any conclusions.

Maybe you're waiting on disk I/O?

What's your iowait under this load?

Cheers

mandric

I don't know never used iowait... I'll look into it. What command do I use?

blender

Your comment from HN:

"I don't know what the bottleneck is really, that is part of the problem. But I know that every now and then when we we get bursts of traffic I can find with a /server-status that 50 apache processing are in a write state."

I think you do know what the problem is ;-)

What is your PHP Error handling and logging set to? How big are your Apache access_log and error_log files?

Cheers

blender

You can use top and look at % "wa" or something like

iostat -x 3

Which will give you lots of disk related info including %iowait

That particular command will give you the stats every 3 seconds until you Ctrl-C out of it.

Cheers

mandric

Well all my log files are rotating normally as far as I know ... here's the global log files. I love logrotate.d.

slice:~# ls -al /var/log/apache2/error.log
-rw-r----- 1 root adm 280975 Apr  2 12:05 /var/log/apache2/error.log
slice:~# ls -al /var/log/apache2/access.log
-rw-r----- 1 root adm 1190895 Apr  2 13:50 /var/log/apache2/access.log

The only bottleneck I have seen is in mysql ... which is hard to debug.

http://andric.us/load/localdomain/localhost.localdomain.html

So you think there is definitely something that is hanging up the apache processes. So my next question is,if it happens again, is there a way to poke apache to figure out what is the processes doing besides /server-status which just tells me they are all in W (writing) mode?

blender

"So you think there is definitely something that is hanging up the apache processes."

Not necessarily.

The overall impression I get from your munin stats is that your bottleneck is the disk.

You do have iowait - look at the pink in the CPU Usage graphs.

High interrupts for blkif - which is some block device driver for Xen I gather.

One thing you might want to look at is temporary tables created on disk by MySQL - this can really hurt an application.

Run this:

mysqladmin -u root -p ext -ri 30 | grep Created_tmp_disk

First run will tell you how many on-disk temporary tables were created since last restart of MySQL. Then it will tell you how many are created in the 30 seconds time window (until you Control-C out of it).

Cheers

mandric

here's the output I let it run for a couple mins

 slice:~# mysqladmin -u root -p ext -ri 30 | grep Created_tmp_disk
 Enter password: 
 | Created_tmp_disk_tables           | 130498      |
 | Created_tmp_disk_tables           | 32          |
 | Created_tmp_disk_tables           | 11          |
 | Created_tmp_disk_tables           | 19          |
 | Created_tmp_disk_tables           | 10          |

blender

I would recommend that you run the MySQL tmpdir on a RAM based file system (e.g. tmpfs)

Note there is a current MySQL bug (http://bugs.mysql.com/bug.php?id=30287) that doesn't honor the tmpdir variable - so make sure you are using a version of MySQL that isn't affected.

Look into what I'm suggesting here - I think it will help you a lot.

Cheers

mandric

So the problem is mysql. http://andric.us/load/localdomain/localhost.localdomain.html

I get these slow queries every now and then, probably from the insane drupal install, and then apache gets hung up. Either because of IO or db. So I can do a mysqlslowqueries ... here's some of the output.

Count: 1 Time=46.00s (46s) Lock=0.00s (0s) Rows=5.0 (5), windyc[windyc]@local host

SELECT * FROM node n LEFT JOIN radioactivity dn ON n.nid = dn.id LEFT JOIN drigg_node dn2 ON dn2.dnid = n.nid WHERE ( N=N AND !(n.type='S' AND dn2.killed =N ) ) AND ( n.type='S' AND status=N AND promote=N ) ORDER BY dn.energy DESC LIMIT N

I will look into the radioactivity module then on one of the drupal installs i guess.

blender

Well, I think it's pretty clear it is I/O as a result of the DB. Look at the load average too - it pretty much mirrors the high interrupts on the blkif device.

The MySQL tmpdir on tmpfs will help but it is treating the symptom - you are right to investigate the queries.

http://dev.mysql.com/doc/refman/5.0/en/internal-temporary-tables.html

We run Drupal too and it is highly relational. Another thing you might consider to speed it up is use the memcache module and get the cache tables out of the DB.

Cheers

Cheers

mandric

Yep sounds good. Unfortunately I'm running etch so ...

slice:~# apt-cache policy mysql-server
mysql-server:
  Installed: 5.0.32-7etch1
  Candidate: 5.0.32-7etch8
  Version table:
     5.0.51a-24~bpo40+1 0
          1 http://www.backports.org etch-backports/main Packages
     5.0.32-7etch8 0
        500 http://ftp.us.debian.org etch/main Packages
        500 http://security.debian.org etch/updates/main Packages
 *** 5.0.32-7etch1 0
        100 /var/lib/dpkg/status

I may have to go with a backport or move off etch to get the mysql tmpfs patch you mentioned.

alangiv

You can get MySQL 5.1.32 from http://www.dotdeb.org/ Just add the appropriate repository to Apt. I installed it on Ubuntu 8.04 without any problems. MySQL, Apache, etc not being updated via the regular repositories sure is a real pain.

mandric

Are there any issues with upgrading from mysql 5.0 to 5.1? Just a minor rev right, my dbs won't break?

alangiv

I had no issue upgrading from 5.0 to 5.1. I backed up all user databases and the system databases just for safety, also expecting that I would need to restore them afterward, however they were upgraded in-place.

After configuring the correct repositories from dotdeb.org, I ran "apt-get -y install mysql-server-5.1" and had no problems at all. Not bad considering other people have had issues with upgrading Ubuntu specifically - the repositories were designed for Etch and Lenny.

blender

According to the MySQL bug it "has been introduced in version 5.0.44"

So you should be okay with your current MySQL.

Cheers

mandric

Hah! Sorry for the noise ... ok back to tmpfs and memcache then!

mandric

blender, so you mentioned "Another thing you might consider to speed it up is use the memcache module and get the cache tables out of the DB", I just want to clarify what you meant by this. Am I right in thinking that I can configure mysql to use memcached as its tmpfs rather than disk? what did you mean by "memcache module"? I assumed http://danga.com/memcached/ .

blender

Sorry, by memcache module I meant the Drupal memcache module:

http://drupal.org/project/memcache

You do need to run memcached so that the Drupal module can use it for the cache tables (i.e. cache, cache_block, cache_filter, etc.) instead of MySQL.

Cheers

mandric

Cool thanks, will keep you posted...

mandric

So for now going with ...


     # mount -t ramfs ramfs /mysql_ramfs/
     # chown mysql:mysql /mysql_ramfs/
     # grep tmp /etc/mysql/my.cnf 
     tmpdir		= /mysql_ramfs
     tmp_table_size = 256M

I just haven't added the ramfs mount to my fstab ... looking into it.

mandric

Well that didn't work, I started getting bunch of 403 errors from the drupal install. I will try again later ... but I found one thread on drupal that explains how to do a ramdisk on osx, just need to find a similar recipe for debian.

http://drupal.org/node/369204#comment-1274786

blender

TMPFS seems to be the best practice when used with MySQL for this purpose. cf:

http://www.bigdbahead.com/?p=121

Maybe give it size=512M instead of the size=2048M in the article to start.

Note that tmp_table_size is limited by size of max_heap_table_size which you have set to 128 (yes, I saw your post on groups.drupal.org ;-) You should've posted that to High Performance group instead of MySQL - you still can if you want a second opinion, you won't hurt my feelings :-) )

Cheers

mandric

Wow this article looks great ... (still reading)

Also here's my post on the mysql list ... which is only gathering dust. The complete silence makes me feel like I'm on the wrong track.

http://lists.mysql.com/mysql/217219

mandric

went ahead and followed the bigdbahead article and mysql took it just fine. monitoring the server, but looks promising so far.

http://andric.us/load/

Thanks again for your help, blender.

Categories