Moving my Server with less than a second downtime

As I’ve written in my previous post, I moved to a new Hetzner Box. This is how I setup my virtual Host and how I moved my server, including all virtual machines, to the new data center with a downtime of a second or less. This setup is only possible if hetzner allows you to move you network via datacenter.


  • Use host routing to avoid wasting IPs
  • install centos-release-qemu-ev to install kvm/qemu from SigVirt/OVirt/RHEV
  • With the newer kvm/qemu you can use virsh migrate --live --copy-storage-all

Don’t waste IP addresses

Because fixed IP addresses are expensive wasting two of the eight IPs to have a broadcast network is not an option.

Virtual Machine setup

We create virtual Interfaces vif0 to vif7 and configure the routing to which we connect the virtual machines on eth0. We let libvirt call a little script on startup of the virtual machines. To do this we add the configurations to the libvirt/quemu xml config of the VMs with virsh edit virt0:

<interface type='ethernet'>
      <mac address='XX:XX:XX:XX:XX:XX'/>
      <script path='/etc/libvirt/scripts/vif-route'/>
      <target dev='vif0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

which calls:


# $1 is interface

case "$1" in
      ip r a xx.xx.xx.x0 dev $1 &> /dev/null
      ip r a xx.xx.xx.x1 dev $1 &> /dev/null
      ip r a xx.xx.xx.x2 dev $1 &> /dev/null
      ip r a xx.xx.xx.x3 dev $1 &> /dev/null
      ip r a xx.xx.xx.x4 dev $1 &> /dev/null
      ip r a xx.xx.xx.x5 dev $1 &> /dev/null
      ip r a xx.xx.xx.x6 dev $1 &> /dev/null
      ip r a xx.xx.xx.x7 dev $1 &> /dev/null
echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp

exit 0

Setup inside the virtual Machine

Now we configure the network and routing for the virtual machines in /etc/sysconfig/network-scripts/ It should be noted, that the IP address is arbitrary. We need it becouse we cannot add a default route directly to a device, in this case eth0.




XX.XX.XX.YY/ZZ dev eth0
default via XX.XX.XX.YY dev eth0


Now we want to move the VM to our new server.

The old server, let’s call it Hetzner-Old, is a CentOS 7 with a bridged net/29 network and several virtual machines running on it (e.g. this Blog). The VMs are running on LVMs which themself are running on crypto devices. Each VM has its own LVM group.

The new server, Hetzner-New is also a CentOS 7 machine. The LVM devices have the same name and are of the same size or bigger. They have to have the same names. This is also a good opportunity to make an LVM device bigger without the need to shut down the according VM.

Now we add the repo for SIG Virtualisation on both machines.

yum install centos-release-qemu-ev

and update qemu/kvm etc

Next we open a tunnel between the two machines with

ssh -w 2:2 tun2-IP

and add ipaddesess on both sides (I will use ifconfig because othwise you have to use two commands)

ifconfig tun2 192.168.0.X/24

This way we have encrytion and don’t have to worry encryting it with TLS on qemu.

Server Diagramm

On Hetzner old we have to route through the tunnel tun2

ip r a net/29 via tun2-IP

Next we do

echo 0 | tee /proc/sys/net/ip4/conf/*/rp_filter

That way, when one of the network interfaces of a VM (vif+) gets removed, when the VM is moved, the according route is removed. Otherwise it would try to go through the default GW of Hetzner-New, wich is not allowed by Hetzner yet. Now, when the VM starts on Hetzner-New it has no interface to this IP-Address and it’s routed through the tunnel, through the GW of Hetzner-Old.

On Hetzner-New we add a new routing table with:

echo 201 vm-out >> /etc/iproute2/rt_tables

The number 201 is only internally used, therefore arbitrary.

Next we set a rule for every package where fwmark 1 is set use table vm-out:

ip rul a fwmark 1 table vm-out

Now we set a new default GATEWAY

ip r a default via IP-Hetzner-Old table vm-out

and set a firewall rule (vif+ are the interfaces for the VMs and iptables uses + instead of the commonly used *

iptables -t mangle -s net/29 -A PREROUTING -j MARK --set-mark 1 -i vif+

Now we are ready for moving the VMs from Hetzner-Old to Hetzner-New

virsh migrate --live VIRTUALMACHINE qemu+ssh://IP-HETZNER-Old/system --copy-storage-all --verbose --persistent

Finally we wait for Hetzner to swich the net/29 network over to Hetzner-New and throw away the previously set iptables rule. Since the new machine is in a different data center, Hetzner has to allow this moving. Because for this to work the net/29 network has to be in Hetzners core routers. With this command we wait for the first tcp package to arrive at Hetzner-New and flush our iptables rule.

tcpdump -i br0 net net/29 -s 1 && iptables -F PREROUTING -t mangle

Posted in Fedora | Tagged | Comments Off on Moving my Server with less than a second downtime

lmsensors for Monitoring a Hetzner EX41 with nct6775 on Centos7


I recently rented a new Hetzner Box to replace my old. I moved from EX40 to EX41 and saving even some money every month :-).

Every thing went smooth, but the sensors did not work. I had to use nct6775, but the module from Centos7 said “No Device”. ElRepo to the rescue. They have a nct6775 kmod available, but this module is not compatible with Centos7.4 :-(.

First I had to created a fixed package. I changed only the release-number and the kversion to  693.2.2 instead of 327. The fixed version is available at: I also open a ticket with elrepo: I don’t know how they will fix this. ElRepo uses kABI-tracking kmods and things like that should not happen. We will see.

With this fixed package installed I created the following configuration file (/etc/sensors.d/hetzner) with the help of sensors -u:

chip "nct6792-*"
 label in0 "2V"
 label in2 "3.3V1"
 label in3 "3.3V2"
 label in6 "1V1"
 label in7 "3.3V3"
 label in8 "3.3V4"
 label in9 "1V2"
 label in12 "1.1V"
 label fan1 "fan"

 set in0_min 2 * 0.80
 set in0_max 2 * 1.2
 # Can not set alarms
 #set in0_alarm: 0.000

 set temp1_max 40
 set temp1_max_hyst 38
 set temp2_max 40
 set temp2_max_hyst 38
 #set temp3_input 40
 # PECI Agent 0
 #set temp7_max 40

 ignore in1
 ignore in4
 ignore in5
 ignore in10
 ignore in11
 ignore in13
 ignore in14
 ignore fan2
 ignore temp4
 ignore temp5
 ignore temp6
 ignore temp8
 ignore temp9
 ignore temp10

 ignore intrusion0
 ignore intrusion1
 ignore beep_enable

I can not fix the alarm for 2V (in0), but the rest looks good. Don’t forget to set the setting with sensors -s.

Update 15.10.2017: The module kmod-nct6775 is now update in elrepo directly, so only the configuration file is needed.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Comments Off on lmsensors for Monitoring a Hetzner EX41 with nct6775 on Centos7

FreeNAS and check_mk


I’m setting up two FreeNAS Server for Backup and Archiving and I really like FreeNAS 11. Thank good I didn’t have time to update it to FreeNAS Coral. 🙂

But I’m using check_mk for monitoring and I would like to use it to monitor FreeNAS as well. There is a check_mk agent for FreeBSD so the only problem is to run it.

I created this script to run it as a Init/Shutdown Script (both pre-init and post-init) . It will create everything you need, only define the BASEDIR at the beginning and put the check_mk_agent for FreeBSD in this directory. Make sure this script (check_mk_setup) and check_mk_agent are executable.

You also need to make sure inetd is running. I enable tftpd for that. Maybe some other service are possible as well. But I only tested it with tftpd.

if grep checkmk /conf/base/etc/inetd.conf &> /dev/null
  echo checkmk stream tcp nowait root $BASEDIR/check_mk_agent check_mk_agent >> /conf/base/etc/inetd.conf

if grep checkmk /conf/base/etc/services &> /dev/null
  echo "checkmk 6556/tcp #check_mk" >> /conf/base/etc/services

if grep checkmk /etc/services &> /dev/null
  echo "checkmk 6556/tcp #check_mk" >> /etc/services

killall -1 inetd

After the next reboot the system can be monitored by check_mk. It even survived the upgrade from FreeNAS 10 to 11.

Posted in Linux, Uncategorized | Tagged | Comments Off on FreeNAS and check_mk

qemu/kvm libvirt and trim with Fedora 25


after more then 10 years of using VMWare Workstation (Starting with VMWare Workstation 5). I’m in the process of moving to KVM/libvirt, but I want to use qcow2 with trim support.

I’m using Fedora 25 with virt-manager to create my virtual machines. A lot of pages describe that very well like Chris Irwins. But I found another problem.

To support trim you need to make sure you have at least 2.1, but I want the latest version 2.7. This it the default, so normally you don’t need to change this with virsh edit DOMAIN:

<type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>

One parameter that you need to change manually (and that can not be done with virt-manager) tells qemu that the discard/trim should be forwarded to the underlying image.It looks like this:

<driver name='qemu' type='qcow2' discard='unmap'/>

All of this can be found on the Internet. A problem that I faced with RHEL7 and others is that virt-manager creates the disc-controller as a antique LSI/NCR controller and that RHEL7 does not support this. To fix this you have to this model to the scsi controller:

<controller type='scsi' index='0' model='virtio-scsi'>

With this RHEL7, Windows 10 and FreeBSD 11 Machines can be configured to run with trimming there images them self.

To migrate you Windows 10 from “libvirt” auf “libvirt-scsi” is quite easy. Add a new libvirt-scsi Disc to an existing Installation. Install the driver for libvirt-scsi, reboot. Now you system supports this driver. After the this reboot, shutdown the machines again and change the disc type of all discs to libvirt-scsi like above and reboot again. Your system should start up with it’s libvirt-scsi enabled disc.

To run the trim command to cleanup unused space, run this in an Admin PowerShell:

Optimize-Volume -DriveLetter c -ReTrim -Verbose

Rinse and repeat for all disc letters.

As always no warranty that this does not break you system.

Posted in Fedora, Linux, Uncategorized | Tagged | Comments Off on qemu/kvm libvirt and trim with Fedora 25

Citrix Reciever and SELinux

Hi Internet,

sorry for the longtime being absent from this blog. But marriage and a child takes time and the blog was the first to go. But I will restart writing blog posts today with a project I started a couple of days ago.

Selinux and the Citrix Reciever

I have to use the Citrix Reciever to access the Citrix farm in our company. This is the only way to access the company network remotely. But I don’t like to run it as unconfined_t on my Fedora 25. So I sat down and created a selinux modules to limit the access of this close source software on my system. In the process I found that it tries to read the mozilla profile and other stuff that I didn’t like and I therefore disabled this. The code is available on GitHub. Simply install the citrix reciever with the rpm from the Citrix website.

This is only a fast and dirty solution. If you want to clean in up, I look forward to it. If I have time I will clean in up, but maybe someone is faster then me (aka. has more sparetime) 😉

I will not give ANY waranty. If it breaks for you it is your problem.

After the installation of Citirx you can run it like this.

sudo dnf install make selinux-policy-devel
git clone
cd selinux-citrixrecievermakesudo 
make load
sudo restorecon -Rv /opt/Citrix/

It works for me, if you run into any problems tell me. I have a dontaudit rule against accessing mozilla_home_t. It still runs perfectly for me.

Next things on my selinux list is tlp-thinkpad. To put selinux rule in for tlp to run with akmod-acpi_call from the TLP Website.

Posted in Fedora, Linux | Tagged , | Comments Off on Citrix Reciever and SELinux

We need 64bit, everywhere!

Just bought a new 8TB disk drive. Following my standard procedures I run a badblock -w against the disk as burn in test. Running on Fedora 20 on x86_64 I was surprised to see this:

badblocks -v -v -w /dev/sdf
 badblocks: Value too large for defined data type invalid end block (7516192768): must be 32-bit value


lvcreate -L 1G --type thin-pool --thinpool thin_pool $VG
lvcreate -T $VG/thin_pool -V 4T -n thinvol
badblocks -v -v -w /dev/$VG/thin_pool


badblock -v -v -w -b 4096 /dev/$VG/thin_pool
Posted in Enterprise Linux, Fedora, Linux | Tagged | Comments Off on We need 64bit, everywhere!

my T-Shirt

root is god
selinux is blasphemy

Posted in Enterprise Linux, Fedora, Linux | Tagged | 2 Comments

yum vs. dnf

Sorry for the long silence. I will try to post more often from now on.
I playing around with dnf, which is the package manager that should replace yum in the future and had some very nice insight.

[root@mysystem ~]# date ; yum clean all
Do 25. Sep 17:51:06 CEST 2014
Geladene Plugins: auto-update-debuginfo, fastestmirror, langpacks, ps, refresh-
                : packagekit, remove-with-leaves, rpm-warm-cache, show-leaves,
                : versionlock
Quellen werden aufgeräumt:adobe-linux-x86_64 fedora fedora-HandBrake
                        : fedora-chromium-stable logstash-1.4
                        : mbooth-eclipse-luna rpmfusion-free
                        : rpmfusion-free-updates rpmfusion-nonfree
                        : rpmfusion-nonfree-updates updates
Cleaning up everything
[root@mysystem ~]# date ; yum update
Do 25. Sep 17:51:13 CEST 2014
Geladene Plugins: auto-update-debuginfo, fastestmirror, langpacks, ps, refresh-
                : packagekit, remove-with-leaves, rpm-warm-cache, show-leaves,
                : versionlock
adobe-linux-x86_64                                          |  951 B  00:00     
fedora/20/x86_64/metalink                                   |  33 kB  00:00     
fedora                                                      | 3.8 kB  00:00     
fedora-HandBrake                                            | 2.9 kB  00:00     
fedora-chromium-stable                                      | 3.4 kB  00:00     
logstash-1.4                                                | 2.5 kB  00:00     
mbooth-eclipse-luna                                         | 3.0 kB  00:00     
rpmfusion-free                                              | 3.3 kB  00:00     
rpmfusion-free-updates                                      | 3.3 kB  00:00     
rpmfusion-nonfree                                           | 3.3 kB  00:00     
rpmfusion-nonfree-updates                                   | 3.3 kB  00:00     
updates/20/x86_64/metalink                                  |  15 kB  00:00     
updates                                                     | 4.9 kB  00:00     
(1/8): fedora-HandBrake/20/x86_64/primary_db                |  10 kB  00:00     
(2/8): fedora/20/x86_64/group_gz                            | 394 kB  00:00     
(3/8): mbooth-eclipse-luna/20/x86_64/primary_db             | 148 kB  00:00     
(4/8): updates/20/x86_64/group_gz                           | 395 kB  00:00     
updates/20/x86_64/primary_db   FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
updates/20/x86_64/primary_db   FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
(5/8): logstash-1.4/primary_db                              |  17 kB  00:00     
(6/8): fedora-chromium-stable/20/x86_64/primary_db          |  29 kB  00:00     
(7/8): updates/20/x86_64/primary_db                         |  12 MB  00:04     
(8/8): fedora/20/x86_64/primary_db                          |  18 MB  00:04     
(1/11): rpmfusion-free-updates/20/x86_64/group_gz           | 1.6 kB  00:00     
(2/11): rpmfusion-free-updates/20/x86_64/primary_db         | 348 kB  00:00     
(3/11): rpmfusion-nonfree/20/x86_64/group_gz                | 1.0 kB  00:00     
(4/11): rpmfusion-nonfree-updates/20/x86_64/group_gz        | 1.0 kB  00:00     
(5/11): rpmfusion-nonfree/20/x86_64/primary_db              | 137 kB  00:00     
(6/11): rpmfusion-nonfree-updates/20/x86_64/primary_db      | 107 kB  00:00     
updates/20/x86_64/updateinfo   FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
(7/11): rpmfusion-free/20/x86_64/group_gz                   | 1.6 kB  00:00     
updates/20/x86_64/pkgtags      FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
(8/11): rpmfusion-free/20/x86_64/primary_db                 | 445 kB  00:00     
updates/20/x86_64/updateinfo   FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
updates/20/x86_64/pkgtags      FAILED                                  [Errno 14] HTTP Error 404 - Not Found
Anderer Spiegelserver wird versucht.
(9/11): adobe-linux-x86_64/primary                          | 1.2 kB  00:00     
(10/11): updates/20/x86_64/updateinfo                       | 1.4 MB  00:00     
(11/11): updates/20/x86_64/pkgtags                          | 1.2 MB  00:00     
Determining fastest mirrors
 * fedora:
 * rpmfusion-free:
 * rpmfusion-free-updates:
 * rpmfusion-nonfree:
 * rpmfusion-nonfree-updates:
 * updates:
adobe-linux-x86_64                                                          2/2
No packages marked for update

So, yum tells me no update available.

[root@mysystem ~]# date ; dnf clean all
Do 25. Sep 17:51:06 CEST 2014
Quellen werden aufgeräumt:logstash-1.4 mbooth-eclipse-luna
                        : rpmfusion-free-updates fedora-HandBrake
                        : rpmfusion-nonfree-updates rpmfusion-free
                        : adobe-linux-x86_64 fedora rpmfusion-nonfree
                        : fedora-chromium-stable updates
Alles wird aufgeräumt
[root@mysystem ~]# date ; dnf update 
Do 25. Sep 17:51:13 CEST 2014
logstash repository for 1.4.x packages        251 kB/s | 364 kB     00:01    
The eclipse-luna Software Collection for Fedo 6.9 MB/s | 388 kB     00:00    
RPM Fusion for Fedora 20 - Free - Updates     6.2 MB/s | 389 kB     00:00    
Open source multiplatform video transcoder     58 kB/s | 6.5 kB     00:00    
RPM Fusion for Fedora 20 - Nonfree - Updates  3.1 MB/s | 106 kB     00:00    
RPM Fusion for Fedora 20 - Free               7.1 MB/s | 487 kB     00:00    
Adobe Systems Incorporated                    3.1 kB/s | 1.8 kB     00:00    
Fedora 20 - x86_64                            3.1 MB/s |  36 MB     00:11    
RPM Fusion for Fedora 20 - Nonfree            5.9 MB/s | 289 kB     00:00    
Builds of the "stable" tag of the Chromium We  82 kB/s | 241 kB     00:02    
Fedora 20 - x86_64 - Updates                   10 MB/s |  26 MB     00:02    
Abhängigkeiten sind aufgelöst.
Installieren    2 Packages
Aktualisieren  80 Packages
Entfernen       4 Packages

and dnf tells me my updates are ready to install! Guest I should start using dnf from now on.

Posted in Fedora, Linux, Uncategorized | Tagged | 3 Comments

Bachelor Thesis: Centralized and structured log file analysis with Open Source and Free Software tools

After a lot of hard work I finished my bachelor thesis end of August and gave my colloquium end of September. Because of all the other stuff going on in my life I just have now time to upload my thesis.

I’m interested in log file analysis for a long time, but in the last years a lot has happened in this area. Here the abstract:

This thesis gives an overview on the Open Source and Free Software tools available for a centralized and structured log file analysis. This includes the tools to convert unstructured logs into structured log and different possibilities to transport this log to a central analyzing and storage station. The different storage and analyzing tools will be introduced, as well as the different web front ends to be used by the system administrator. At the end different tool chains will be introduced, that are well tested in this field.

Because of the time delay, some infos are already dated, but only two things, as fas as I know:

1. Graylog2 has now a new version available in Beta.

2. logstash has release a new version 1.2.1

I will keep this updated over time here on the blog, so have a look from time to time. If you find any problem, please inform me here at the blog or via mail:  “logfiles” (add the @ at the right place).

The URL of the Thesis is at:

Have a log of fun and tell me what you think.

Posted in Enterprise Linux, Fedora, Linux, logs | Tagged , | 1 Comment

note to self: Fedora minimal Installation and virt-manager

If you need to install virt-manager on a Fedora/Red Hat minimal installation, add the following packages:

yum install xorg-x11-fonts-Type1 xorg-x11-xauth libvirt-daemon-kvm libvirt virt-manager qemu-kvm qemu-kvm-tools
Posted in Uncategorized | Tagged | Comments Off on note to self: Fedora minimal Installation and virt-manager