So you have 262772k used swap .. that's a lot and will likely cause a noticeable slowdown.
Also I'd like to point out that the 22% memory usage per ruby process is 22% of what all programs are using .. so might be more than 22% of actual RAM .. if you get my drift .. so it'll always add up to 100% .. but might be 250% of RAM.
I'd look at using less ruby processes .. if you're using 'mongrels' for deployment .. maybe look at using fusion passenger.
This link has a lot of good info.
Also if you can rebuild to a 32 bit Slice, that might help a bit .. 32 bit generally uses a little less RAM. NB: 'rebuild' == wipe the Slice clean, including any rails installations..