Fixing DRBD “Lower device is already claimed”
In my adventures with DRBD I recently ran across this error that took me a couple days to resolve:
d(data) 1: Failure: (114) Lower device is already claimed. This usually means it is mounted.
[data] cmd /sbin/drbdsetup 1 disk /dev/sdb /dev/sdb internal –set-defaults –create-device –on-io-error=detach –no-disk-barrier –no-disk-flushes failed – continuing!
I was a bit confused since the filesystem wasn’t mounted anywhere (mount showed only the system volumes) and there weren’t any other programs using the raw disk.
Creating a filesystem also ran into a similar error:
/dev/sdb is apparently in use by the system; will not make a filesystem here
Obviously this is not just a problem with drbd.
I eventually stumbled across a reference to /dev/dm-0, something that had recently popped up on both my DRBD boxes. A quick search showed that this is related to multipathd.
dmsetup ls VG_XenStorage--xxx--xxx--xxx--xxx--xxx-MGT (253, 0)dmsetup status VG_XenStorage--xxx--xxx--xxx--xxx--xxx-MGT: 0 8192 linear
Ah, go figure, Xen talked to the multipathd service and convinced it to cause problems for me. Fixing this problem is as easy as editing (or creating) /etc/multipath.conf and inserting the following:
blacklist {devnode'*' }
As it turns out, this is not the solution (but is probably not a bad idea anyways since I don’t use multipathing). After rebooting the problem persisted which lead me to more googling.
I ran across this thread on the OpenFiler forums suggesting that it may be an issue with LVM which lead me to this thread on the Citrix forums confirming this. Basically XenServer uses LVM to manage it’s storage repositories and if LVM on the storage host system finds and mounts these partitions before DRBD can get started there will be trouble.
To resolve this, edit /etc/lvm/lvm.conf and add a filter= statement similar to the following
filter =[ "r|/dev/sdb|", "r|/dev/sda|" ]
This tells LVM to ignore /dev/sdb and /dev/sda when scanning for logical volumes, in this case I have drbd resources on both sda and sdb and would prefer that LVM keep its grubby little mitts off both disks.
After making the change reboot your servers and all should be well with the world.