Not signed in (Sign In)
    •  
      CommentAuthornatebeaty
    • CommentTimeMay 9th 2008
     permalink
    Hello all..

    I recently got a notice that my slice was swapping heavily after 8 months of moderate but steadily increasing traffic. I have since fixed the problem, but am wondering if there are still more optimizations I can do to keep things running smoothly (and deal with increasing traffic).

    I'm on a 1gb slice, primarily hosting a handcoded PHP/MySQL site that gets ~3mil pageviews a month, ~6,500 uniques a day. I also host 4 low-traffic (50 uniques a day on the busiest) Django sites using apache2/mod_python, *and* a handful of also low-traffic PHP sites.

    To alleviate the swapping I set up nginx to serve static files before hitting apache2, which was incredibly easy and immediately effective. I also found some heavy SQL queries using the slow log and optimized where I could, which helped a lot. (The busy PHP site's traffic is primarily punbb, and mods for PM'ing and Feedback were PHP- and SQL-challenged, oof.) I might set up Django to be served with nginx and fastcgi at some point, which would free up some server memory, I'm impressed with nginx so far -- it's incredibly fast and has a light footprint.

    top currently shows:

    top - 12:23:58 up 10 days, 43 min, 1 user, load average: 0.57, 0.40, 0.43
    Tasks: 86 total, 2 running, 84 sleeping, 0 stopped, 0 zombie
    Cpu(s): 2.4%us, 1.3%sy, 0.0%ni, 94.2%id, 1.9%wa, 0.0%hi, 0.1%si, 0.2%st
    Mem: 1047296k total, 1002328k used, 44968k free, 54576k buffers
    Swap: 2097144k total, 62916k used, 2034228k free, 199564k cached

    7377 mysql 15 0 239m 176m 5216 S 5 17.3 462:33.08 mysqld
    1894 clamav 16 0 139m 88m 1004 S 0 8.7 3:03.98 clamd
    32297 www-data 16 0 168m 33m 7172 S 0 3.3 0:05.38 apache2
    32295 www-data 16 0 165m 31m 7208 S 1 3.1 0:05.15 apache2
    32306 www-data 17 0 120m 14m 4728 S 2 1.4 0:03.46 apache2
    32296 www-data 16 0 120m 14m 4768 S 2 1.4 0:04.86 apache2
    32165 www-data 17 0 120m 14m 4724 S 1 1.4 0:08.14 apache2
    32337 www-data 16 0 120m 14m 4624 S 0 1.4 0:01.86 apache2
    32317 www-data 16 0 119m 14m 4592 S 2 1.4 0:02.54 apache2
    32294 www-data 16 0 119m 14m 4760 S 1 1.4 0:04.32 apache2
    32348 www-data 16 0 119m 13m 4548 S 2 1.4 0:00.63 apache2
    4643 root 16 0 116m 13m 7440 S 0 1.3 0:01.57 apache2

    apache2 often gets up to 50+mb from the Django sites, and they aren't recycled as quickly as they used to be when apache2 was handling all the traffic, but things have been much more stable and I haven't swapped at all since installing nginx. I used to have mysql using much less memory, but I've realized some punbb queries require large chunks of memory, so I bumped it up. (Can post relevant my.cnf if requested..)

    I guess I'm wondering if .4 - .9 load avg is OK for this amount of traffic and if MySQL seems to be using a sane amount of memory. I'm only a part-time sysadmin by necessity, so any help is much appreciated!

    ps: SliceHost RULES! heh.
    •  
      CommentAuthorSuperJared
    • CommentTimeMay 9th 2008
     permalink
    Two things to help recycle apache processes: disable KeepAlive (keep it on in Nginx, though), and set MaxRequestsPerChild to 1.
    •  
      CommentAuthornatebeaty
    • CommentTimeMay 9th 2008
     permalink
    Awesome, thanks for the feedback, I'll give that a shot.

    This will certainly free up memory, as there are usually 7 or more 30+mb apache2 instances and a few at 50mb. I notice my top dump above looks more healthy than normal!

    I guess I just didn't know if it was healthier to leave processes running or to cycle apache on a regular basis..
    • CommentAuthorartagesw
    • CommentTimeMay 9th 2008 edited
     permalink

    Definitely disable KeepAlive on Apache. But MaxRequestsPerChild should be greater than 1 in my opinion. Forcing Apache to create and destroy a child process for every request is going to make a high-traffic site less responsive. Setting it to 100 or so should be fine on a low-traffic site. With as much traffic you are seeing, you could go higher – 1000 or more – unless your PHP code is leaking tons of memory. More critical are your settings for ServerLimit and MaxClients.

    •  
      CommentAuthornatebeaty
    • CommentTimeMay 9th 2008
     permalink

    I've been monitoring my slice with KeepAlive Off and MaxRequestsPerChild 1, and I haven't seen an improvement in memory or load avg. If anything, they're a little worse.

    Some relevant apache.conf settings:

    #KeepAlive On
    KeepAlive Off
    MaxKeepAliveRequests 100
    KeepAliveTimeout 4
    
    <IfModule prefork.c>
    StartServers       5
    MinSpareServers    5
    MaxSpareServers    10
    MaxClients         40
    # MaxRequestsPerChild 250
    MaxRequestsPerChild 1
    </IfModule>
    

    I've never set the ServerLimit directive, but have experimented with many different other prefork directives before nginx calmed the apache2 madness, heh. From apache docs:

    With the prefork MPM, use this directive only if you need to set MaxClients higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxClients to.

    • CommentAuthorilikepi
    • CommentTimeMay 9th 2008
     permalink
    In the general sense, I don't think there's anything wrong with load averages of less than number-of-cpus. You still have some idle time occurring in that case. It looks like much of your busy CPU time is going to mysql, which seems logical for the usage you describe. Personally, I would think a MaxRequestsPerChild of 1 would actually increase load average due to the overhead of constantly spawning and cleaning up httpd processes.
    •  
      CommentAuthornatebeaty
    • CommentTimeMay 9th 2008
     permalink

    That definitely seemed to be the case. I guess there's a balance to be had here, as when I bump up MaxRequestsPerChild to 1000, I am back to having 8 apache2 processes with 30-50mb each and only around 10mb of free server memory with load avg around .8 -- if I set MaxRequestsPerChild back to 250 I have 200mb of free memory and load avg around .5. The server is not swapping using either one, however.

    I don't believe this is from PHP memory leak as apache was steady at ~14mb before I moved over my Django sites and added mod_python. I'll try moving these to nginx-fastcgi.

    I'm still seeing queries of 2-3 seconds in my slow log every few minutes, but using EXPLAIN they seem to be optimized with the correct indices (I could definitely be missing something). Is this normal for a site with heavy usage of punbb forums?

    • CommentAuthorartagesw
    • CommentTimeMay 9th 2008
     permalink

    If your apache processes are indeed growing to 50MB, you don’t want MaxClients set to 40 on a 1GB slice. If all 40 processes are called into service (during a period of heavy traffic), you’re going to need 2GB RAM just for Apache. Obviously you are going to swap like mad in that scenario.

    If you can limit the growth of your apache processes by lowering MaxRequestsPerChild, then you can have a somewhat higher value for MaxClients. But I would not go over 15 MaxClients or so given what you have shown us.

    Also, lowering MaxSpareServers to 5 will free up RAM for other purposes when Apache doesn’t need it.

    •  
      CommentAuthornatebeaty
    • CommentTimeMay 10th 2008
     permalink

    @artagesw: thanks for your help, those changes seemed to have stabilized load around .3 - .6, and recycles the apache processes enough to keep memory open enough for no swapping with fluctuations in traffic. I feel like I've been stabbing in the dark, slowly getting a grasp on these settings, and this helped me understand them quite a bit.

    Interesting that when you have Django sites on your slice, you should limit your MaxClients to less than what's normally recommended. But throwing nginx into the mix to alleviate apache from the mundane task of dishing out static files seems to play a big factor in making this possible.

    •  
      CommentAuthorcactus
    • CommentTimeMay 10th 2008
     permalink

    I bet you would see a marked improvement from moving django instances to fcgi+nginx.
    Apache will not need to pull in the mod_python framework on each process fork, so the per child memory usage will go down.

    As for mysql, you might check to see if you have the query cache enabled. For sites that do lots of simple reads, it can be a big help (with the tradeoff of a little extra memory being used). Of course, do some measurements to see if it is really being beneficial or not.

    As for tuning apache, I also recommend disabling keepalives. However, I would keep the maxrequestperchild value higher than 1 (you seemed to have decent results with 250).

    If I were in your shoes, I would probably turn on mod_info for a localhost virtualhost, and start collecting some metrics. Take a look at the max and min children over the course of a day, to give youself some ideas of how many spare servers to start with, what the max values you would normally see, etc. I personally use collectd to gather that data, and make my own graphs with rrdtool against the rrds that are output.

    Good luck!

    :D

    •  
      CommentAuthormandric
    • CommentTimeOct 13th 2008
     permalink

    Use monin for monitoring so you can figure out what is eating up your resources.