Not signed in (Sign In)

Ruby + Rails: Memory usage

Bottom of Page

1 to 20 of 28

  1. <
  2. 1
  3. 2
  4. >
    • CommentAuthorgobigdave
    • CommentTimeMay 1st 2008
     permalink
    I have a couple of CentOS 5.1 slices here plus one CentOS 5.0 server at ServerBeach. I installed all three (plus a fourth here at the office) using the same set of instructions, but the slices all use double the memory. My ServerBeach (SB) server s my current production server, but I'm want to move it to a slice. If I run top on the SB server, all the mongrel processes use about 50M of RAM. After thousands of page views, that might jump to 55-57M. The same was true for my server I built in the office. However, both my slices start at 50M for the mongrel processes, but after < 10 page views that is up to 100M. What's the deal? It makes it difficult for me to move my stuff over because I can't run as many mongrel instances on my slice.

    FYI, I compile ruby 1.8.6 from sources at ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6.tar.gz. Everything else (apache, rails, etc.) is all doe via yum or gem. Again, I used the exact same procedure for all four servers, but only the slices use up a bunch of memory.

    Is there a library I need to replace before building ruby? What is the deal?
    • CommentAuthorSchultz
    • CommentTimeMay 1st 2008 edited
     permalink

    What version of CentOS at ServerBeach do use as in x86 or amd64? As the 64bit version of binarys always use more memory then 32bit versions. All of slicehosts distros are 64bit versions. But besides that it still seems 100M memory is still alot compared to the others.

    Good post on this
    http://forum.slicehost.com/comments.php?DiscussionID=1275&page=1#Item_0

    • CommentAuthorgobigdave
    • CommentTimeMay 1st 2008
     permalink
    The server at SB is an AMD machine with CentOS 5.0, and I'm pretty sure it's not running 64bit. The other office machine is an old Intel P4 with CentOS 5.0 (definitely 32 bit). Both these machines consume half as much memory as my two slices. As a result, they perform better even though their CPUs are weaker. Something definitely doesn't seem right here. On my 256M slice, I can barely run one mongrel instance (I'm swapping even with one instance).

    Based on the link you posted, it looks like I may need to rethink moving everything to SliceHost. 64bit doesn't do me any good, and I'm guessing 64bit doesn't do anyone using a 256 (or even 512) slice any good either. All you're doing is wasting memory.
    • CommentAuthorilikepi
    • CommentTimeMay 1st 2008 edited
     permalink

    What number are you looking at in top? I’m running several mongrel instances on a CentOS 4 slice. They all claim 100 MB of virtual memory, but most only claim ~50 MB resident. The RadiantCMS site that gets hit fairly often (maybe 10 views/min peak?) has around 70 MB resident.

    edit: it’s a 1024slice, fwiw

    • CommentAuthorgobigdave
    • CommentTimeMay 1st 2008
     permalink
    I'm looking at resident number. As of a minute ago, my test/staging server with no traffic is sitting at: 193M VIRT, 95M RES.

    I wonder if I can change my slice to be 32 bit. I don't see what good 64 bit does for you until you have more than 2 gig of RAM. If anything, 64 bit would slow you down with less memory.
    • CommentAuthorgobigdave
    • CommentTimeMay 1st 2008
     permalink
    OK, I just checked the files and confirmed that ruby on slicehost is 64bit, but it is 32bit on my ServerBeach server. I'm guessing that is the main difference.

    This seems like a huge waste to me. Other than ego, 64bit does you no good unless you have more than 2-3Gig of RAM. Why burden smaller slices with it?

    Can anyone point me to a procedure to setup the necessary libraries to get the 32bit versions of these apps installed? Otherwise, I was better off with my crappy 5 year-old P4 because it could have more running.
    • CommentAuthorgobigdave
    • CommentTimeMay 1st 2008
     permalink
    For comparison, on my 32bit server with 1Gig of RAM I can easily have 6 mongrels, mysql, apache, a background rails daemon, and starling running without swapping. I can service thousands of pages without hardly noticing. On my 512M slice, I can barely get 2 mongrels going with the same configuration. If I extrapolate out, I would probably need to go the 2 Gig slice to get the same number of mongrels running without swapping. 1 Gig might make it, but I'm guessing I would be swapping since just the mongrels would be over 700M. Unfortunately, a 2Gig slice is almost double what I'm paying now for a 1Gig server. Again, with < 2 Gig RAM, 64bit is not helping.
    • CommentAuthorSchultz
    • CommentTimeMay 2nd 2008 edited
     permalink
    I am not sure why you are seeing such high number. I am using Gentoo here on slicehost. Installed Mephisto blog using 3 mongrel instances and i configured nginx to proxy to the 3 instances. used ab to do 1000 requests with 3 concurrent along with hitting pages up to "warm" up the mongrel instances and i max out right now at 56m usage

     7678 mongrel  15   0  161m  56m 3644 S    0 22.1   0:03.10 /usr/bin/ruby18 /usr/bin/mongrel_rails start -d -e production -a 127
     7675 mongrel  15   0  160m  56m 3652 S    0 22.0   0:03.37 /usr/bin/ruby18 /usr/bin/mongrel_rails start -d -e production -a 127
     7672 mongrel  15   0  160m  55m 3636 S    0 21.7   0:02.92 /usr/bin/ruby18 /usr/bin/mongrel_rails start -d -e production -a 127

                 total       used       free     shared    buffers     cached
    Mem:           256        245         10          0          1         34
    -/+ buffers/cache:        209         46
    Swap:          511         25        486

    Also with running nginx with 3 worker processes, PHP with 3 worker processes and 3 mongrel instances along with running mysql. I still have 46megs of RAM left. This is all on a 256 64bit Gentoo slice. Not sure why CentOS is so high memory usage.
    • CommentAuthorilikepi
    • CommentTimeMay 2nd 2008 edited
     permalink

    On my CentOS 5 512slice, I was able to compile a 32-bit ruby binary after first installing glibc-devel.i386 with yum. I had to export CFLAGS=-m32 and LDFLAGS=-m32 before running configure. I also added —target=i686-unknown-linux-gnu, but I’m not sure if that was necessary. The result is this:

    $ file ruby ruby: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2. 6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped $ ./ruby --version ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux]
    Might be worth trying it with your app to compare the memory usage. Realistically, comparing memory usage of your app to mine or anyone elses isn’t too scientific…

    •  
      CommentAuthorsesser
    • CommentTimeMay 2nd 2008
     permalink

    I'm running mephisto via one mongrel and 3 php-cgi processes behind nginx (2 processes) as well as mysql. My mongrel process is currently at ~62M res/135M virtual. I don't really think 64bit architecture is your problem. Is your app on Slicehost running in development mode?

    • CommentAuthorgobigdave
    • CommentTimeMay 2nd 2008
     permalink
    sesser, I have two slices. One is running with RAILS_ENV=staging and the other is RAILS_ENV=production. Both environment files (config/staging.rb and config/production.rb) are the same. I have two production machines: one on Serverbeach and one slice. Other than a different target IP, both are deployed exactly the same. Unless there is some setting I'm missing to not run in production mode, I'm pretty sure I'm running in production mode.

    ilikepi, I'm only comparing my app one two different servers. Comparing to others is more of a ballpark figure. Thanks for the how-to on building ruby 32bit, but I must be missing something. I get ruby compiled correctly, but mongrel won't fire up. I get:

    /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/http11.so: /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/http11.so: wrong ELF class: ELFCLASS64 - /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/http11.so (LoadError)
    from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
    from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel.rb:12
    from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
    from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
    from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/mongrel_rails:11
    from /usr/local/bin/mongrel_rails:19:in `load'
    from /usr/local/bin/mongrel_rails:19

    It looks like I need to redo mongrel as well?
    • CommentAuthorSchultz
    • CommentTimeMay 2nd 2008
     permalink
    Yea you will need to recompile mongrel as 32bit. As you can't have 32bit code load 64bit libraries and you can't have 64bit code load 32bit libraries
    • CommentAuthorgobigdave
    • CommentTimeMay 2nd 2008
     permalink
    I figure that was the issue. Now, I need dig into gem install to figure out how to do that. I have a feeling, however, that I may need to simply accept it. Although, I still question the need for 64 bit for any slice under 2gig.
    • CommentAuthorilikepi
    • CommentTimeMay 2nd 2008
     permalink

    Hmm…with my fresh install, mongrel’s http11.so library is already 32-bit:
    $ file usr/lib/ruby/gems/1.8/gems/mongrel-1.1.4/lib/http11.so usr/lib/ruby/gems/1.8/gems/mongrel-1.1.4/lib/http11.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped
    I wonder if that’s because I still had CFLAGS/LDFLAGS=-m32 exported when I ran gem install, or because gem is somehow configured to build in 32-bit instead of 64…‘gem environment’ lists this:
    - RUBYGEMS PLATFORMS: - ruby - x86-linux

    • CommentAuthorSchultz
    • CommentTimeMay 2nd 2008
     permalink
    Posted By: gobigdaveI figure that was the issue. Now, I need dig into gem install to figure out how to do that. I have a feeling, however, that I may need to simply accept it. Although, I still question the need for 64 bit for any slice under 2gig.

    Its because they do run more then 2gigs of ram probably on the main server that hosts the VPS's and the requirement for Xen is this
    Posted By: slamb
    I believe you’re right that with Xen paravirtualization, the domU and dom0 architecture must match. But you can definitely use i386 userspace binaries inside an x86_64 kernel (Xen or not), and that’s what I was referring to. In fact, I have a mix on my slice now.

    That was from the post i linked.
    •  
      CommentAuthorsesser
    • CommentTimeMay 2nd 2008
     permalink

    The memory limit for 32bit architecture is 4GB (3.5GB if you're running a less than superior OS), not 2... just, FYI :)

    • CommentAuthorgobigdave
    • CommentTimeMay 2nd 2008
     permalink
    True, but most systems still limit processes to 2Gig. That's the limit I was referring to. I haven't run into yet, but don't most linux systems require some extra config to address more than 2-3 gig as well? Then there is Windows (definitely less than superior). It only lets you access 2Gig period without a registry hack. I'm so happy not to be in that world anymore.
    • CommentAuthorilikepi
    • CommentTimeMay 3rd 2008
     permalink

    Yeah on x86, Linux requires you to enable large memory support in the kernel if you have more than a gig. I suspect on x86_64 that configuration option is ignored since it’s pointless…

    • CommentAuthorgobigdave
    • CommentTimeMay 4th 2008
     permalink
    I decided to give 64 bit a go for a while, and I was not pleased. On my 512M slice, I have 2 mongrels, apache, mysql, and one background process. It's the exact same setup I had on my 1 Gig ServerBeach server (6 mongrels over there though). After running for 48 hours, my swap size was up to 328M, and the site's performance was suffering. Never had any swapping on the 32bit machine. I keep hearing of people running 2 mongrels on a 256M slice. Why am I using so much memory here but not in the 32 bit distro?

    Looks like I need to go back and figure out how to get mongrel and my gems to compile in 32 bit. I wasn't able to do that before.
    • CommentAuthorilikepi
    • CommentTimeMay 5th 2008 edited
     permalink

    When I tested this, I just compiled as a regular user and installed under my home directory. This is pretty painless…all you need is something like ‘—prefix=${HOME}/usr’ passed to the configure script for ruby. Just make sure you have ‘${HOME}/usr/bin’ in your PATH variable before anything else (and also the other stuff I talked about in previous posts, like CFLAGS/LDFLAGS and perhaps the ‘—target’ parameter, and also glibc-devel.i386 installed on the system).

    edit: I should mention I had a problem with Rails 2.0.2 and the openssl external module from Ruby. On our production web server, I ran into the same problem when I upgraded to 2.0.2: mongrel refused to start because I hadn’t enabled ext/openssl and ext/digest/md5 when I first installed Ruby. After I enabled them and recompiled Ruby, mongrel was happy again. When I was playing with compiling for 32-bit on the new server, I also encountered this problem (because I forgot to enable those extensions). However, when I tried to enable them, ext/openssl wouldn’t compile. It wasn’t throwing an error, the Makefile simply wasn’t issuing any compile commands. I have no idea why…I have both versions of the openssl and openssl-devel packages installed. Rather than trying to diagnose this, I just downgraded to Rails 1.2.6, since it didn’t really matter for my basic test. If your app requires Rails 2.0.2…you may run into this. Let me know and I’ll go back and poke it again (which a bigger stick, perhaps).

1 to 20 of 28

  1. <
  2. 1
  3. 2
  4. >