Not signed in (Sign In)
  1.  permalink

    I have a Django Python 256 Slice that has a Nginx front end which serves up static media files, passes Django/Python requests back to an Apache 2.2 prefork MPM server utilizing mod_wsgi. I have two small, low-traffic sites on the slice.

    Yesterday I had to restart the slice because the swap was being eaten up. Page load times were 10 seconds plus. Today the memory usage already seams to be spirling out of control. Here are some images of memory usage:

    free

    top

    Apache seams to be eating memory like there's no tomorrow. Does anyone have any suggestions to help keep Apache memory usage down?

    Here's my apache Config:

    Timeout 300
    KeepAlive Off
    MaxKeepAliveRequests 100
    KeepAliveTimeout 15
    <IfModule mpm_prefork_module>
        StartServers          5
        MinSpareServers       5
        MaxSpareServers      10
        MaxClients          150
        MaxRequestsPerChild   0
    </IfModule>
    

    Edit: changed category.

    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    I'm not seeing that swap is out of control.

    Both free and top are showing that used swap is zero.

    If you wanted to reduce the memory footprint of apache you could move to mpm-worker or reduce the number of start servers to something like 2 (don't forget to adjust the other numbers accordingly - info is in the apache2.conf file).
  2.  permalink

    I know they swap's not being used right now. I said they it was being used yesterday, before I restarted the server.

    Currently Apache is using 60%+ memory for two sites that are lucky to get more then 1500 hits a month combind.

    How can I make the swap from mpm-prefork to mpm-worker?

    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    assuming you are using ubuntu (note, carefullly read over the remove plan to make sure it is not removing any dependencies that you are currently using in your apps):

    $ sudo aptitude remove apache2-mpm-prefork
    $ sudo aptitude install apache2-mpm-worker apache2-threaded-dev

    That should do it.
    Thankful People: brownelltyler
  3.  permalink

    I'm getting an error with that install command:

    The following NEW packages will be installed:
      apache2-mpm-worker
    The following packages will be REMOVED:
      apache2-mpm-prefork
    0 packages upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
    Need to get 251kB of archives. After unpacking 8192B will be used.
    Do you want to continue? [Y/n/?] Y
    Writing extended state information... Done
    Err http://archive.ubuntu.com hardy-updates/main apache2-mpm-worker 2.2.8-1ubuntu0.3
      404 Not Found [IP: 91.189.88.46 80]
    E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2-mpm-worker_2.2.8-1ubuntu0.3_amd64.deb: 404 Not Found [IP: 91.189.88.46 80]
    E: Unable to correct for unavailable packages
    
    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    1. Is your system still up and running - can you still see your web apps?
    2. Try running

    $ sudo aptitude safe-upgrade
    $ sudo aptitude full-upgrade

    3. Try the install again
  4.  permalink
    1. Yes, my apps are still up and working.
    2. Both of these commands didn't install anything.
    3. Same error as before.
    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    I have a meeting to run off to, but I will look into this when I come back. I'm glad to hear that your server is still running. It looks like the ubuntu archive server is down, or that the address changed (which aptitude upgrade-full should have taken care of).
    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    I searched for your issue on google and found a few resources. I would advise you look through them to find your specific problem. I can't vouch for a particular solution as I have not run into this issue myself.

    http://www.google.com/search?rlz=1C1GGLS_enUS291US306&sourceid=chrome&ie=UTF-8&q=404+Not+Found+[IP:+91.189.88.46+80]
    Thankful People: brownelltyler
  5.  permalink

    This worked:

    sudo dpkg --purge -a
    sudo apt-get update
    

    The install worked. I'm up and running with mpm-worker now.
    Since I'm not very familiar with with mpm-worker, at all, is the defaut configuration overkill for a small server? Does anyone have a recommended "sweet spot"?

    <IfModule mpm_worker_module>
        StartServers          2
        MaxClients          150
        MinSpareThreads      25
        MaxSpareThreads      75
        ThreadsPerChild      25
        MaxRequestsPerChild   0
    </IfModule>
    

    Memory usage is way down and the sites are preforming noticabley faster. They seam to have hickups every 5th page load or so, taking 3-5 seconds to load. The normal page load it hovering around 250ms now, so that's nice.

    Can the swap from prefork to worker be done on a server using PHP as well? Or a server using any scripting language? If prefork is that much of a memory hog then why is it the default to begin with?

    • CommentAuthormeppum
    • CommentTimeMar 20th 2009 edited
     permalink
    Glad to hear everything worked out.

    As for the sweet spot, it depends on your system resources, the webapps themselves, and the load you expect. Though, the default settings should be more than enough for what you have described. If you want to dial them down feel free to do a little research.

    I'm not 100% sure, but I think that using worker for PHP is ill advised unless you are using something like fastcgi to generate your php pages.

    Prefork is the default because it is safer. Specifically, it is "more" threadsafe than worker because each request is run in its own process.
  6.  permalink

    I ask because I have a LAMP slice that I know notice is using swap.

    free

    top

    Here's the configuration for that server:

    Timeout 30
    KeepAlive On
    MaxKeepAliveRequests 200
    KeepAliveTimeout 2
    <IfModule mpm_prefork_module>
        StartServers          5
        MinSpareServers       5
        MaxSpareServers      10
        MaxClients          175
        MaxRequestsPerChild   0
    </IfModule>
    

    Thoughts?

    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    Well, my first question is what is the load on that server? If it is low, I would consider lowering your maxclients setting.
  7.  permalink

    Very low. I'll cut it down to 100.

    • CommentAuthormeppum
    • CommentTimeMar 20th 2009
     permalink
    consider something like this

    StartServers 1
    MinSpareServers 1
    MaxSpareServers 3
    MaxClients 50
    Thankful People: brownelltyler
  8.  permalink

    Considered and implemented. Thank you. I'll monitor it over the next while and adjust accordingly.
    Thanks again for your time!

    • CommentAuthordrobbins
    • CommentTimeMar 20th 2009
     permalink
    With very low settings for start/min/maxservers, Apache with spend a lot of time spawning new ones and then killing them off. Which will probably translate into significantly more CPU usage. You want to find a balance for those settings, so that you aren't running an unnecessary number of Apache threads (consuming memory needlessly) but at the same time you aren't forcing Apache into a very frequent spawn/kill cycle (using a lot of CPU and creating load).

    You should also look at the MaxRequestsPerChild settings, and perhaps set it to something other than 0 (unlimited). An Apache process grows in memory size to the size required, but it never shrinks in size. So if a process serves monsterpage.php it will grow to the large memory size required for that page and stay there. MaxRequestsPerChild gives you some control over this. Set it to say 500 and each Apache process will get killed off once it has served 500 requests. So that bloated process that served monsterpage.php will eventually die and Apache will spawn a fresh one, using much less memory, when needed.

    Depending on your traffic you might experiment with MaxRequestsPerChild set to 500 - 2000. Generally, a higher setting for a busier Apache server.
    Thankful People: brownelltyler
  9.  permalink

    Excellent. It's one thing to read the manual about how these work, and another to have it explained in plain English.
    It's a really low traffic server, so I'll set MaxRequestsPerChild to 500 for now.

    Thanks!