As I’ve written in my previous post, I moved to a new Hetzner Box. This is how I setup my virtual Host and how I moved my server, including all virtual machines, to the new data center with a downtime of a second or less. This setup is only possible if hetzner allows you to move you network via datacenter.
- Use host routing to avoid wasting IPs
centos-release-qemu-evto install kvm/qemu from SigVirt/OVirt/RHEV
- With the newer kvm/qemu you can use
virsh migrate --live --copy-storage-all
Don’t waste IP addresses
Because fixed IP addresses are expensive wasting two of the eight IPs to have a broadcast network is not an option.
Virtual Machine setup
We create virtual Interfaces vif0 to vif7 and configure the routing to which we connect the virtual machines on eth0. We let libvirt call a little script on startup of the virtual machines. To do this we add the configurations to the libvirt/quemu xml config of the VMs with
virsh edit virt0:
<interface type='ethernet'> <mac address='XX:XX:XX:XX:XX:XX'/> <script path='/etc/libvirt/scripts/vif-route'/> <target dev='vif0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
#!/bin/bash #/etc/libvirt/scripts/vif-route # $1 is interface case "$1" in vif0) ip r a xx.xx.xx.x0 dev $1 &> /dev/null ;; vif1) ip r a xx.xx.xx.x1 dev $1 &> /dev/null ;; vif2) ip r a xx.xx.xx.x2 dev $1 &> /dev/null ;; vif3) ip r a xx.xx.xx.x3 dev $1 &> /dev/null ;; vif4) ip r a xx.xx.xx.x4 dev $1 &> /dev/null ;; vif5) ip r a xx.xx.xx.x5 dev $1 &> /dev/null ;; vif6) ip r a xx.xx.xx.x6 dev $1 &> /dev/null ;; vif7) ip r a xx.xx.xx.x7 dev $1 &> /dev/null ;; esac echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp exit 0
Setup inside the virtual Machine
Now we configure the network and routing for the virtual machines in
/etc/sysconfig/network-scripts/ It should be noted, that the IP address is arbitrary. We need it becouse we cannot add a default route directly to a device, in this case eth0.
/etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0" BOOTPROTO="static" HWADDR=XX:XX:XX:XX:XX:XX IPV6INIT="no" IPV6_AUTOCONF="no" NM_CONTROLLED="no" ONBOOT="yes" IPADDR=XX.XX.XX.YY NETMASK=255.255.255.255 TYPE=Ethernet /etc/sysconfig/network-scripts/route-eth0 XX.XX.XX.YY/ZZ dev eth0 default via XX.XX.XX.YY dev eth0
Now we want to move the VM to our new server.
The old server, let’s call it Hetzner-Old, is a CentOS 7 with a bridged net/29 network and several virtual machines running on it (e.g. this Blog). The VMs are running on LVMs which themself are running on crypto devices. Each VM has its own LVM group.
The new server, Hetzner-New is also a CentOS 7 machine. The LVM devices have the same name and are of the same size or bigger. They have to have the same names. This is also a good opportunity to make an LVM device bigger without the need to shut down the according VM.
Now we add the repo for SIG Virtualisation on both machines.
yum install centos-release-qemu-ev
and update qemu/kvm etc
Next we open a tunnel between the two machines with
ssh -w 2:2 tun2-IP
and add ipaddesess on both sides (I will use ifconfig because othwise you have to use two commands)
ifconfig tun2 192.168.0.X/24
This way we have encrytion and don’t have to worry encryting it with TLS on qemu.
On Hetzner old we have to route through the tunnel tun2
ip r a net/29 via tun2-IP
Next we do
echo 0 | tee /proc/sys/net/ip4/conf/*/rp_filter
That way, when one of the network interfaces of a VM (vif+) gets removed, when the VM is moved, the according route is removed. Otherwise it would try to go through the default GW of Hetzner-New, wich is not allowed by Hetzner yet. Now, when the VM starts on Hetzner-New it has no interface to this IP-Address and it’s routed through the tunnel, through the GW of Hetzner-Old.
On Hetzner-New we add a new routing table with:
echo 201 vm-out >> /etc/iproute2/rt_tables
The number 201 is only internally used, therefore arbitrary.
Next we set a rule for every package where fwmark 1 is set use table vm-out:
ip rul a fwmark 1 table vm-out
Now we set a new default GATEWAY
ip r a default via IP-Hetzner-Old table vm-out
and set a firewall rule (vif+ are the interfaces for the VMs and iptables uses + instead of the commonly used *
iptables -t mangle -s net/29 -A PREROUTING -j MARK --set-mark 1 -i vif+
Now we are ready for moving the VMs from Hetzner-Old to Hetzner-New
virsh migrate --live VIRTUALMACHINE qemu+ssh://IP-HETZNER-Old/system --copy-storage-all --verbose --persistent
Finally we wait for Hetzner to swich the net/29 network over to Hetzner-New and throw away the previously set iptables rule. Since the new machine is in a different data center, Hetzner has to allow this moving. Because for this to work the net/29 network has to be in Hetzners core routers. With this command we wait for the first tcp package to arrive at Hetzner-New and flush our iptables rule.
tcpdump -i br0 net net/29 -s 1 && iptables -F PREROUTING -t mangle