Ceph libvirt locking

When you store your virtual machines in ceph with libvirt, you have the problem that any server could boot the same virtual machine at the same time, which would result in a broken filesystem. Fortunately, ceph provides a way to lock a rbd image, and libvirt provides a hook to execute a script, that can rescue yourself from this horrible situation. I've written the following shellscript, you have to place it in /etc/libvirt/hooks/qemu, make it executable and restart libvirtd afterwards.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
#!/bin/bash
# wogri@wogri.com

# this hook script creates a lock on a ceph rbd image when a VM is started. It also handles live migration smoothly with locks.
# It also will complain and not start a VM when a rbd image is tried to be attached more than one time to a VM.

domain=$1
command=$2

function lock {
  rbd=$1
  locktype=$2
  echo locking rbd image with command: rbd lock add $rbd $locktype --shared libvirt >&2
  rbd lock add $rbd $locktype --shared libvirt
}

function unlock {
  rbd=$1
  locktype=$2
  client=$(rbd lock list $rbd | grep -v 'Lock tag:' | grep $locktype | awk '{ print $1 }')
  echo unlocking rbd image with command: rbd lock remove $rbd $locktype $client >&2
  rbd lock remove $rbd $locktype $client
}

function is_locked {
  rbd=$1
  locktype=$2
  rbd lock list $rbd | grep $locktype > /dev/null
}


# the domain xml comes from STDIN in any case
# we need to use grep here, ase you can't call a virsh command within a virsh hook. sorry folks.
for rbd in $(grep "protocol='rbd'" - | sed "s/^.*name='//" | sed "s/'(\/)?>$//")
do
  if [ $command = 'migrate' ]
  then
    lock $rbd migrate
  elif [ $command = 'prepare' ]
  then
    if is_locked $rbd migrate
    then
      # this means a live migration has been started, and we do not need to re-lock the VM. 
      exit 0
    fi
    if is_locked $rbd libvirt
    then
      echo image $rbd is locked. will not start the VM. Type rbd lock list $rbd for more information. >&2
      lock $rbd startup
      exit 257
    fi
    lock $rbd libvirt
  elif [ $command = 'started' ]
  then
    # let's remove the startup lock, so the release hook can properly unlock the rbd image.  
    if is_locked $rbd startup
    then
      unlock $rbd startup
    fi
  elif [ $command = 'release' ]
  then
    if is_locked $rbd startup
    then
      echo RBD image $rbd seems to be locked, vm could not be started, so the image will stay locked. Type rbd lock list $rbd for more information. >&2
      unlock $rbd startup
    else
      if is_locked $rbd migrate
      then
        # this means a live migration has been finished, and we want to remove the 'migrate' lock, but not the 'real' lock. 
        unlock $rbd migrate
      else
        unlock $rbd libvirt
      fi
    fi
  fi
done
Letzte Änderung: 29.01.2014