Tag Archives: VSAN

ESXi Local Storage Preparation for VSAN

I recently set up a new vSphere lab cluster with the goal of trying out VSAN. My existing lab had been using an old Equallogic SAN for shared storage, and frankly I was pretty tired of the noise. My first plan was to move over to Nutanix Community Edition, so I bought some SSDs for the cache tier on my three hosts, and set up Nutanix. That turned out to be a complete dumpster fire. I really wanted Nutanix CE to be good, I’m a fan of their commercial platform as a host for VMware, and really hoped that they could pull off the hypervisor as well.

Nope. The reason why not is a subject for a separate rant.

Anyway, the other thing I wanted to try was VMware VSAN, and since I already know that vSphere is a great hypervisor platform, I was much more comfortable with this. Setup was an absolute breeze. Once I had vCenter up and running with all my hosts connected, configuring VSAN was a piece of cake. Well, mostly, anyway. I had previously set up Nutanix CE on these drives, and since they already had partitions allocated, the VSAN disk import process didn’t see any of my disks. The solution was to delete the partitions on each of the disks from an SSH session to the host.

To see the partitions, you can simply do
ls /dev/disks/
which should show you something like this:
ls /dev/disks

Notice that there’s an entry there called mpx.vmhba32:C0:T0:L0, and then there’s 6 more after with a :1, :5, etc after it? The entry without the :# at the end is the parent disk, all the others are partitions on the disk, and the number after the colon is the partition number. You can see more details about the partitions by doing
fdisk -l
which will display all the partitions on all the disks on the system. You can display just the disk you are interested in by doing
fdisk -l /dev/disks/mpx.vmhba32:C0:T0:L0
which displays just the partitions for that one disk.
fdisk -l /dev/disks/mpx.vmhba32:C0:T0:L0
Incidentally, the tool fdisk is deprecated, the replacement for this is
partedUtil getptbl /dev/disks/mpx.vmhba32:C0:T0:L0
but I find the output of fdisk -l to be much more readable. Anyway…

How can you tell what these devices are anyway? Try
esxcli storage core device list
which outputs a ton of information about all of the storage devices attached to the host. The relevent snip here is
esxcli storage core device list
You can see from this that this device has the display name “Local USB Direct-Access”, this is the USB flash drive that has ESXi installed on it.

To delete a partition that you don’t want, like a left-over Nutanix CE partition, use
partedUtil delete /dev/disks/DeviceName PartitionNumber
So if I wanted to delete partition 9 on my USB stick (a really bad idea) I’d use the command
partedUtil delete /dev/disks/mpx.vmhba32:C0:T0:L0 9

To allow the VSAN disk import process to detect the disks, go through and delete all the partitions on all the disks you want to use as part of your VSAN volume.

One other problem I ran into was that I set up a VSAN on all my disks, then had to blow away my entire vSphere environment and start from scratch again, reloading the hosts with ESXi and setting up a new vCenter. After this, I tried to set up VSAN and again the disks were not found. I attempted to delete all the partitions on the disks, but it wouldn’t let me do it. Turns out that vCenter saw that those disks had been part of a VSAN disk group before, and so it treated the disks as in-use. The solution was to run the command
esxcli vsan storage list
which gave me info like this for every disk on the host:
esxcli vsan storage list
Then I could run the command
esxcli vsan storage remove -u UUID
where UUID is the VSAN Disk Group UUID listed for the relevent disks. In this example, I would run
esxcli vsan storage remove -u 52709227-18ac-39ba-c7cd-8815f3eeafbe

I believe, but I don’t remember for sure, that I needed to then manually delete the partitions on the disks after this, before the VSAN disk import would recognize them.

Once I did all this, I was able to set up VSAN and create a volume. VSAN is pretty sensitive to the hardware it’s run on, and my hardware is most definitely not on the HCL. But hey, it works well enough for a lab environment.

More to come…

Currently drinking: Jekyll Brewing Southern Juice IPA

Leave a Comment

Filed under VSAN