I have been, for the most part, a very happy Slicehost customer for several months now. The hardware, the people, the services and the pricing have all been really good. That said, there's one _major_ issue that I keep running into with slices here, and it may just drive me over the brink...

All the VPS's run on x86_64 architecture. I have no complaint with that, except that it costs me twice as much money. Actually, I guess that's a big complaint. :)

In order to run all my services, I need a 512MB slice, even though my disk and bandwidth are really low. The reason is that my database servers, perl processes, and httpd processes are all consuming _twice_ as much memory on the 64-bit architecture as they would with a 32-bit architecture. What does that mean to me? Well, basically, I could make do just fine with a 256MB slice, but I have to pay for a 512MB slice. Even worse is that recently I had some problems with really bad performance, and it was suggested by a slicehost staff member in IRC that I should make sure my swap is close to 0. That would be a complete waste of money, but it was the solution offered to me at the time. (As it turns out, another VPS on my server was thrashing _HARD_ on the disk i/o, and that is why my slice slowed to a crawl. The tech folks were great when they figured out what was going on, and had me back at full strength in no time).

To try to combat this memory issue, I've tried "mixing and matching" 64-bit, and 32-bit packages in my CentOS distro. By moving to 32-bit versions of mysqld, httpd, and perl, I've saved TONS of memory (I used to fill up the 512MB slice, plus about 200MB of swap with all 64-bit binaries. Now, all of my services fit into about 300MB of RAM, and I'm using about 50MB of swap). The problem, of course, is that now I have some stuff that's 32-bit, and some that's 64-bit, and I'm running into dependency problems every time I try to do something new. Not to mention that compiling an i386 package on an x86_64 machine is a complete PITA.

All that said, I'm now thinking that maybe I should start shopping around for another VPS provider. One that will put me into a 32-bit distro. This sucks, because I really like Slicehost, and their business model. But how can I justify paying more for something that doesn't buy me anything? None of the apps / servers I have installed are making any use of the 64-bit address space. They're just consuming more memory, and costing me more money.

Does anyone else have any thoughts/comments/feelings on this?

Thanks,
Pat

Posted By: esmInteresting. Actually, there's really no reason you couldn't convert to a "pure" 32-bit userspace, much like you've already started to do. Procedurally, I'd just generate a list of 64-bit RPMs I currently had installed with something like:

rpm -qa --queryformat="%{name}-%{version}.%{arch}\n" | grep 'x86_64$'

Then install 32-bit versions of each, and remove the 64-bit versions. There may be a few you can't get away from, but that'll get you close.

Yes... That's exactly what I've started to do... And this is also why I'm starting to get a little frustrated. :) If all you're doing is running software out of packages, this would probably be fairly easy. But once you need to start compiling custom stuff you get to start arguing with the build systems about architecture, dependencies, libraries, etc. Then you install mock, and argue with building stuff in a chroot... Fun fun. I'm currently in the process of trying to build a mod_python for CentOS 5 that uses python 2.6. Which means that apache and python both need to be 32-bit architectures, and for that, I have to compile python in a mock chroot environment, etc.

All this is pain and frustration I wouldn't have to go through if we could get a 32-bit slice. If the RAM usage difference wasn't more than 2x, I wouldn't care, but we're talking about a big chunk of change here...

All comments and advice are welcome. :)

Actually, most types don't require any more storage on 64 bit architectures - ints are still 4 bytes, for example. Only the size of a pointer needs to change, from 4 bytes to 8. So unless you have a process that allocates huge amounts of RAM used exclusively for storing long lists of pointers, it's not actually going to use twice as much memory.

Want proof? Compile and run the following C program:
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char** argv) {
printf("%d\n", sizeof(int));
printf("%d\n", sizeof(void*));
return 0;
}
:) David

Posted By: diazonaActually, most types don't require any more storage on 64 bit architectures -ints are still 4 bytes, for example. Only the size of a pointer needs to change, from 4 bytes to 8. So unless you have a process that allocates huge amounts of RAM used exclusively for storing long lists of pointers, it's not actually going to use twice as much memory.

Ok... I'm not going to argue with this, because I don't know. :) Here's what I _do_ know: The apps I run on my slice take up significantly more RAM in their 64-bit varients than they do in their 32-bit variants. Want proof? Set up two slices, install favourite distro (I'm using CentOS5), and on one slice, replace as many packages as you can with the 32-bit versions. Watch memory consumption be reduced significantly. :)

Cheers,
Pat

Here's a little more "proof":

pmap -d for my dovecot-auth (x86_64) process:
mapped: 61788K writeable/private: 1404K shared: 0K

pmap -d after removing dovecot.x86_64, and installing dovecot.i386:
mapped: 7596K writeable/private: 964K shared: 0K

Pretty big difference, no? And that's on a relatively small app. Imagine a large perl process, or a DB server?

Regards,
Pat

Okay, Django doesn't count ;-)) as it runs within Python (which itself may not count since it runs within Apache, if you're just using mod_python) . . . and in the previous example with Dovecot, the 64-bit version has more than 8x as much memory mapped - it can't all be accounted for by the change between 32-bit and 64-bit pointers. free shows that you're using 254MB, which would (barely) fit in a 256 MB slice . . . but maybe your programs would use less RAM if they didn't have so much available. Have you tried running the same server setup on a 256 slice to see what happens?

Bottom line, you'll need to decide which is more important to you, a 32-bit architecture or Slicehost's excellent support, management, and business model. That's purely your choice and I don't think there's a whole lot someone like me can do to help you there. (For what it's worth, I'm staying right here at Slicehost)

:) David

I honestly don't believe anyone should (or need) to run an MTA on their slice anymore. Google Apps is free (I've used it for a while now > 1 year) and it's been great.

Mail hosting is a bit of a dark art these days; something that needs to be understood properly (I'm not claiming you do or don't, just that it needs 'dedication').

Also, what are you running in Perl that's so important? Is there not an equivalent that's less hungry?

Who runs Perl these days anyway? :)

I'm with atc: I've managed mail servers of varying complexity since 1994, and finally migrated the email for my personal domain that I've operated for the last 10 years to Google Apps a few months ago. It's just not worth the headache anymore, unless I'm getting paid to do it. ;)

Back to the topic at hand: I really don't see the problem with manually reverting to a 32-bit userland as Pat has done. Yes, it's a little more work up-front, and yes, it'd be nice if Slicehost provided 32-bit userlands by default, but it's nothing you can't fix post-install.

I was running nginx/mongrel to serve rails applications on a 64-bit Debian Etch (on a 256MB slice). I created a 32-bit chroot in which to run 32-bit versions of mongrel, ruby and rails; and, nginx (running outside the chroot in 64-bit land) proxies to the mongrel (in the 32-bit chroot). So it's possible to have programs working with each other in both userlands -- I didn't have to move everything in other words, only what I needed.

The mongrels went from 80MB to 40MB each. Maybe Django users will experience similar memory savings.

Anyway, I documented the whole procedure of getting it set up here.

Posted By: markodesignI'm adding my voice to the chorus here. I'm shocked at how much ram it takes to run a basic ruby web stack on a 64-bit ubuntu 256 slice. We are hitting swap with two instances of ruby running (one production, one dev). I am about to go down the road of recompiling everything as 32-bit, which is going to be a tangent that throws me behind schedule. It would be awesome if Slicehost had a pre-baked solution for this.

you might look into http://www.rubyenterpriseedition.com/ they finally have it working properly under 64bit environments. Which should use less memory for your ruby processes for 64bit.