Overcoming VMWare ESXi’s 2tb limit with Dell PERC 6 and H700/H800 using disk groups with many virtual disks

VMWare ESXi (version 4, I wonder about what ESXi 5 really brings) has a 2TB limit on datastore size.
So say you just bought a new Dell server and disk array, with let’s say 24 500gb disks, and wanna run it in RAID-6 for a total of 10230gb (22 * 465gb).
You’ll soon discover VMWare will limit the datastore to a little less than 2Tb. I’ll not go into the mess that is the reason for this, just accept it. Or buy ESXi 5 for US$ 10k.
One (poor) solution, would be to setup a few smaller RAID arrays (eg, four 5-disk 500gb RAID-5 arrays); each array totalling around 2tb, and create a few datastores, each with a little less than 2tb.
That’s bad because obviously you’ll waste a lot of disks; but it works.
The best solution is to make use of a little-advertised feature of RAID cards: disk groups. Since PERC 6 (PERC 6/i, PERC 6/E, H700 and H800) the card has 3 concepts: Physical Disks, Disk Groups, and Virtual Disks.
When you use OMSA, or go into Ctrl-R during boot, the ‘Create Virtual Disk Wizard’ makes it easy to create a Virtual Disk in a specific RAID mode.
So when you create a new 24 physical disk RAID-6 virtual disk, it automatically creates the Disk Group for you (it’s a ‘wizard’, right?).
What’s not so obvious is that you can put more than one Virtual Disk inside a Disk Group, effectively creating more than one LUN on a RAID array.
To do that, during the first VD create step, reduce the VD “size” field to 2000gb; it will create the first VD in the Disk Group.
Then, repeat the VD creation; it will allow you to re-use the same RAID level and Physical Disks, to create new VDs in the same group.
This way you can create a few VDs (all below 2Tb) without ‘wasting’ physical disks. In VMWare ESXi, create Datastores, one for each VD: set the VMWare block size to 8mb so that you can create large VMWare Virtual Disks in them.
For a large Virtual Machine, you can create many VMWare Virtual Disks (each up to 2tb, each in different datastores) for the VM; inside that VM’s operating system, use the OS’s features to JBOD each virtual disk into a large volume.
You can do that with Windows’s Dynamic Disks, or using Linux software raid md (in JBOD mode): just make sure to use JBOD or other non-stripped mode, because the RAID is actually done and working at the physical controller card Disk Group level.

3 Replies to “Overcoming VMWare ESXi’s 2tb limit with Dell PERC 6 and H700/H800 using disk groups with many virtual disks”

  1. Great information. I am also scratching my head on this one trying to deploy a file server with large graphic files. You answered my question. I may just leave the file server alone.

  2. I’ve followed your instructions but when I go to vSphere both of the new VDs have the same LUN. How do you modify the LUN of the second VD?

Leave a Reply