IPSec with setkey/racoon and multiple single-host SPDs

Suppose you have an IPSec tunnel with two or more single-hosts (instead of, say, a /24 network). Only one host will ping after restarting Racoon. If you ping host A first, you can’t ping host B later, and vice-versa. This has bugged me for a whole morning. I googled for this a lot and had trouble finding anything, but then found http://lists.freebsd.org/pipermail/freebsd-net/2003-November/002002.html and the answer by Helge Oldach at http://lists.freebsd.org/pipermail/freebsd-net/2003-November/002004.html.

Thanks to the KAME people!

Dell DRAC5 Virtual Media CDROM breaks OS installation, hangs on Windows boot, and makes megaraid_sas insane

We bought a dozen shiny new Dell 2950 and 1950 machines, with DRAC5 cards which seem quite handy.

So off we went to install Debian and Windows 2003 using only DRAC5’s virtual Console (kind of an IP KVM) and virtual media. No physical access to the servers is required. It all works very well, you point DRAC at an .ISO image, configure virtual media to attached in the Ctrl-E screen, hit F11 and choose to boot from “VIRTUAL CDROM”. This is where OS Installation, and problems, begin.

When installing Debian, when virtual media is attached, the PERC controller shifts stuff around so that the virtual media device can fit. That makes my RAID-5 array on the PERC be at /dev/sdc instead of the expected /dev/sda. This seems alright at first, but after the installation is done, GRUB goes insane. You can jackhammer GRUB to work, but then the megaraid_sas driver goes insane. Literally. Lockups, wierd and misleading messages. Insanity reigns.

When installing Windows without the help of the Dell Installation CD, it locks up right at the first “Windows Setup” blue screen, even before the “F6” driver prompt. The machine literally freezes (you can still restart it using DRAC, sweet). If you try using the Dell Install CD, it goes okay until the first reboot where the same thing happens. Complete lockup.

After spending a lot of time with Dell Support with clueless people (they were about to send us a tech to swap the PERC 6i controller which was “obviously” the culprit), we finally realized the problem was the Virtual Media in DRAC5. When we disabled Virtual Media and used a real, physical, spinning DVD drive, everything was fine. It turns out Windows locked up because it couldn’t figure out the “VIRTUAL CDROM” thing. Nothing was wrong with the PERC. Debian installed like a dream too, on /dev/sda.

If you really need to use the Virtual Media, you have to go through a very boring process. First, attach virtual media in Ctrl-E screen and mount the ISO in your browser. Start installing your operating system. When it’s time for the first reboot, go into Ctrl-E again and dettach virtual media. Now the install should proceed as normal…

All in all, DRAC5 is a very useful thing, it’s KVM capabilities have already paid for themselves if only in gasoline saved in trips to the datacenter. Now a Firefox 3 plugin, and fixes to the Virtual Media problems, would be very nice. Hello Dell?

Always, always use memtest86

This week my workstation started freezing and BSOD’ing. I ignored it for a few days until today it finally corrupted my registry.
I spent a few hours restoring the registry from a restore point only to stop at a STOP 0x7E or 0x08E bluescreen.
It turns out I skipped the religious memtest86+ (http://www.memtest.org/) check when I bought this machine. Wierd, I always make sure to run it on all my servers, but somehow managed to forget to do it on my workstation. Now, with 1GB less RAM and two days work lost, I think I’ll never forget it again.

Flash is Single Threaded: RPC and AsyncToken behavior

So in Flex you can invoke a service (like a WebService method, or HTTPService.send()), which returns a token (mx.rpc.AsyncToken). After that, you set a responder on the token. When the results come back, the responder is called. Calling the send() method actually sends the request, so the whole construct looks bizarre, since you’re setting a responder/handler AFTER the call has been sent. This is a very common pattern in Cairngorm applications.

The fact is that this never fails, and I wondered why, because it looks possible (although unlikely) for an RPC call to return data before the responder is set. After thinking for a while, I remembered one of my first problems with Flash, years ago: it’s single threaded. After investigating on the web, I found out this is really what makes this (and a whole host of other async stuff) work in Flash/Flex. The call may return data, but since it’s single threaded, your method will surely be able to continue, and set the responder, before the result handler is called. Uncommon, to say the least.

Ultimate Debian Etch VMWare Server setup (16Gb Dual Quad Core Dell 2950 with OMSA)

With assistance from the helpful folks at the VMWare Discussion Forum, I came up with what I think is the ultimate VMWare Server setup. It’s essentially Debian Etch (4.0) x86 running on a Dell 2950, with dual Xeon E5310 (Quad-Core) and 16Gb RAM. The focus here is on cleanliness: no hacks, no bizarre compat libraries, no patches. Just plain Debian-Dell-VMWare fun.
First, a word about x64. Since I have a lot of RAM, the initial tests were with the amd64 port of Debian. Unfortunately, and not being Debian’s fault, a lot has conspired against x64. First, VMWare not being a native 64-bit application, a lot of compat libraries would be involved. Also, a lot of trouble arises with Dell OMSA. Finally, most people agree that there’s some performance penalty for VMWare on x64, quirks aside. If we had native VMWare and OMSA, I’d go with amd64. The best option I’ve found (and recommended on the forums) is installing standard x86 Debian, and then switching kernel to a ‘bigmem’ one. More on that later.
So lets head on to the method:

1) Prepare the RAID setup for initial install. Dell does not support Debian, so you need a way to configure your RAID volumes beforehand (at the very least, the volume where you will install Debian, but I just go ahead and configure all my volumes). For this purpose I’ve found the excellent “CentOS Dell OMSA 5.1 Live CD”. Just burn the ISO, boot from it, set the root password, wait a lot, and you can get into the Web Interface for OMSA where you can set up the storage. I’ve set “Adaptive Read Ahead”, “Write back” and other performance-enabling RAID options.

2) Install Debian Etch (x86). I used a 3-in-1 CD, which has a i486, amd64 and power installer, and with that it was a little tricky to force i386/i486 instead of amd64 (it seems to auto-detect). The Dell PERC controller and NICs were all detected, even though it seems the (Gigabit) Ethernet ports were reversed (port 1 was eth1 and port 2 was eth0 – I’ve seen this reported before). Proceeded with standard install, I went with straight ext3 filesystems. You may want LVM or other fancy stuff, but I didn’t bother. I took care of unchecking the “Standard system” install option, which would install a lot of uneeded stuff like NFS, and results in a very, very minimal Debian install. Complete install, set root password, apt sources etc, and boot into the new system.

3) New kernel and packages. “aptitude update” and install SSH and the new kernel “aptitude install ssh linux-image-2.6-686-bigmem”. This should be all handled automatically, including the new kernel being setup in grub. Reboot. Once the system is back up, check /proc/cpuinfo, /proc/meminfo and “top” for correct 8-logical-CPU and 16635720 kB MemTotal. You may want to do a “aptitude upgrade” too to update everything with security updates. I also like ntpdate, tz-brasil and ntp-server for keeping the clock right.

4) Install Dell OMSA. The nice folks at sara.nl offer some excellent pre-packaged Dell OMSA install for Debian (it’s version 5.2, so it just plain works, without the “storage not found” problem). Thanks to them, you can just add “deb http://ftp.sara.nl/pub/sara-omsa dell sara” to /etc/apt/sources.list (ftp:// also works), do “aptitude update”, and “aptitude install dellomsa” and it’s quite ready. It’s a large package, around 100mb, but installs perfectly. After the install, run “update-rc.d dsm_om_connsvc defaults” and “/etc/init.d/dsm_om_connsvc start”. You can then go to “https://x.y.z.w:1311” (on another machine) and get to the OMSA web interface. Log-in as “root” with root’s shell password. It’s just that sweet. Dell’s own install system on supported OSes are not even close. I will post about SNMP support for OMSA later.

5) Install VMWare prerequisites. VMWare Server links dynamically to a lot of stuff. I got a few pointers from other sites, but ultimately “ldd” and “apt-cache search” were my best friends. The best indication you’re missing libraries is the VMWare install program complaining about them, or even rejecting your VMWare serial number, or vmware-mui telling you vmware isn’t installed. Anyway, this is the final aptitude line for install all the dependencies I found: “aptitude install libx11-6 libx11-dev libxtst6 xinetd wget linux-headers-`uname -r` build-essential gcc g++ psmisc libxt6 libxrender1 libxi6”.

6) Install VMWare Server and VMWare Management Interface. You get these from vmware.com after registering. Don’t forget to request your free serial number from them too. You should download the latest version (I got 1.0.3) in .tar.gz format. Just “tar xzvf” them, and run the installer script as per the manual. Everything should go smooth – VMWare will say you’ll need to compile the modules, and that should go just fine (no need for any-to-any patches). Do the same install routine for vmware-mui. Once it’s installed you can get to the nice VMWare web interface via “https://x.y.z.w:8333”. You will also connect via VMWare Console (I run that on Windows…) on x.y.z.w port 902. VMWare is able to see around 14.5Gb of RAM, which is excellent for my needs. Any given VM is limited to 3600mb, as usual.