With assistance from the helpful folks at the VMWare Discussion Forum, I came up with what I think is the ultimate VMWare Server setup. It’s essentially Debian Etch (4.0) x86 running on a Dell 2950, with dual Xeon E5310 (Quad-Core) and 16Gb RAM. The focus here is on cleanliness: no hacks, no bizarre compat libraries, no patches. Just plain Debian-Dell-VMWare fun.
First, a word about x64. Since I have a lot of RAM, the initial tests were with the amd64 port of Debian. Unfortunately, and not being Debian’s fault, a lot has conspired against x64. First, VMWare not being a native 64-bit application, a lot of compat libraries would be involved. Also, a lot of trouble arises with Dell OMSA. Finally, most people agree that there’s some performance penalty for VMWare on x64, quirks aside. If we had native VMWare and OMSA, I’d go with amd64. The best option I’ve found (and recommended on the forums) is installing standard x86 Debian, and then switching kernel to a ‘bigmem’ one. More on that later.
So lets head on to the method:

1) Prepare the RAID setup for initial install. Dell does not support Debian, so you need a way to configure your RAID volumes beforehand (at the very least, the volume where you will install Debian, but I just go ahead and configure all my volumes). For this purpose I’ve found the excellent “CentOS Dell OMSA 5.1 Live CD”. Just burn the ISO, boot from it, set the root password, wait a lot, and you can get into the Web Interface for OMSA where you can set up the storage. I’ve set “Adaptive Read Ahead”, “Write back” and other performance-enabling RAID options.

2) Install Debian Etch (x86). I used a 3-in-1 CD, which has a i486, amd64 and power installer, and with that it was a little tricky to force i386/i486 instead of amd64 (it seems to auto-detect). The Dell PERC controller and NICs were all detected, even though it seems the (Gigabit) Ethernet ports were reversed (port 1 was eth1 and port 2 was eth0 – I’ve seen this reported before). Proceeded with standard install, I went with straight ext3 filesystems. You may want LVM or other fancy stuff, but I didn’t bother. I took care of unchecking the “Standard system” install option, which would install a lot of uneeded stuff like NFS, and results in a very, very minimal Debian install. Complete install, set root password, apt sources etc, and boot into the new system.

3) New kernel and packages. “aptitude update” and install SSH and the new kernel “aptitude install ssh linux-image-2.6-686-bigmem”. This should be all handled automatically, including the new kernel being setup in grub. Reboot. Once the system is back up, check /proc/cpuinfo, /proc/meminfo and “top” for correct 8-logical-CPU and 16635720 kB MemTotal. You may want to do a “aptitude upgrade” too to update everything with security updates. I also like ntpdate, tz-brasil and ntp-server for keeping the clock right.

4) Install Dell OMSA. The nice folks at sara.nl offer some excellent pre-packaged Dell OMSA install for Debian (it’s version 5.2, so it just plain works, without the “storage not found” problem). Thanks to them, you can just add “deb http://ftp.sara.nl/pub/sara-omsa dell sara” to /etc/apt/sources.list (ftp:// also works), do “aptitude update”, and “aptitude install dellomsa” and it’s quite ready. It’s a large package, around 100mb, but installs perfectly. After the install, run “update-rc.d dsm_om_connsvc defaults” and “/etc/init.d/dsm_om_connsvc start”. You can then go to “https://x.y.z.w:1311″ (on another machine) and get to the OMSA web interface. Log-in as “root” with root’s shell password. It’s just that sweet. Dell’s own install system on supported OSes are not even close. I will post about SNMP support for OMSA later.

5) Install VMWare prerequisites. VMWare Server links dynamically to a lot of stuff. I got a few pointers from other sites, but ultimately “ldd” and “apt-cache search” were my best friends. The best indication you’re missing libraries is the VMWare install program complaining about them, or even rejecting your VMWare serial number, or vmware-mui telling you vmware isn’t installed. Anyway, this is the final aptitude line for install all the dependencies I found: “aptitude install libx11-6 libx11-dev libxtst6 xinetd wget linux-headers-`uname -r` build-essential gcc g++ psmisc libxt6 libxrender1 libxi6″.

6) Install VMWare Server and VMWare Management Interface. You get these from vmware.com after registering. Don’t forget to request your free serial number from them too. You should download the latest version (I got 1.0.3) in .tar.gz format. Just “tar xzvf” them, and run the installer script as per the manual. Everything should go smooth – VMWare will say you’ll need to compile the modules, and that should go just fine (no need for any-to-any patches). Do the same install routine for vmware-mui. Once it’s installed you can get to the nice VMWare web interface via “https://x.y.z.w:8333″. You will also connect via VMWare Console (I run that on Windows…) on x.y.z.w port 902. VMWare is able to see around 14.5Gb of RAM, which is excellent for my needs. Any given VM is limited to 3600mb, as usual.