Slicehost Forum - Slice starts swapping, runs out of memory, and dies... once or twice a month

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-05T00:45:02+00:00

Ok, i've been having a issue for some time (about 7 months) with my slice where it goes down occasionally - about once a month. It still accepts http requests, but they just time out. Can't login via ssh, same timeout problem. Only way to get it back up is a hard reboot.

When I have a look in the slicehost console, I see messages about the slice being out of memory and killing either mysql or apache2 (seems to vary). What I can't work out is what is causing it to run out of memory - never actually managed to run top in this state. However, I think it happens quickly because when I had munin running (uninstalled it now) I didn't see a gradual increase in ram usage before it crashes - still 300-400mb swap free on the last time munin refreshes before the crash.

I got a kernel log showing it doing the killing, don't think it has any clues in it as to what's taking up the ram though: http://pastebin.com/f3393c4bc

My apache mpm settings: (My sites are all quite low traffic)

StartServers 3
MinSpareServers 1
MaxSpareServers 3
MaxClients 50
MaxRequestsPerChild 1000

I'm using mod_php. I've tried switching to nginx + fastcgi, but a few of the older sites hosted on my server use php in .html files so that doesn't seem to be a option.

My slice had 6 hours of downtime yesterday because it went down while i was asleep - it'd be really nice to finally fix this.

Thanks
Jordan

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-05T14:48:37+00:00

Jordan,

These links may provide some assistance in tuning MySQL and Apache appropriately for your slice. I would recommend that you review them.

http://chrisjohnston.org/2008/configuring-a-lightweight-apache-mysql-install-on-debian-ubuntu

http://www.onlamp.com/pub/a/onlamp/2004/02/05/lamp_tuning.html

-Mark

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-06T04:23:24+00:00

Addition the links Mark gave, you may check here for the apache tuning:http://httpd.apache.org/docs/2.0/misc/perf-tuning.html

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-06T05:28:30+00:00

Ok, I have tuned my apache and mysql based on those sites, but it seems to me that the settings I had would run less servers than recommended than those, so how could increasing the number of servers to start stop me running out of ram? Unfortunately it's really hard to know if the problem's been fixed because it only happens occasionally...

My ram usage normally is and has always been quite low with at max say 50-100mb swap used and not with many writes/reads. It does seem to be something that happens suddenly, not a gradual gain in ram usage.

Also, when my slice has crashed and I go to slice stats, it says i have 100% cpu usage - which is very strange? Also a bit bad because that can't be good for the other slices on the server :<.

Thanks
Jordan

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-08T16:52:32+00:00

a MaxClients setting of 50 seems high if you have a 256MB slice... if each client is around 15MB (as it is one mine), that could be a problem.

On some distros the allowed PHP memory size in php.ini is set extraordinarily high (>100MB)... it could be that a poorly written PHP script is consuming all the memory...

You can also use the apache "ab" program to simulate the load and see if/when it swaps.

Slice starts swapping, runs out of memory, and dies... once or twice a month

2009-04-08T22:26:13+00:00

Well, my php memory limit was set to 128mb - so that does seem entirely plusable that it is a poorly written script (the biggest site on the server is my dad's, and some of his code is ... functional). I've reduced it to 20mb which seems more reasonable.

I guess all i can do now is wait and see if it happens again... thanks a lot for the help.