Firewalld Fedora 34 -> 35 Masquerade between Zones not working anymore

I updated my firewall from 34 to 35 and my firewall was not working anymore. There is a not good documented change with the release of firewalld 1.0 that hit me.

The fix is simple if you found it.

firewall-cmd --permanent --new-policy policy_int_to_ext
firewall-cmd --permanent --policy policy_int_to_ext --add-ingress-zone public
firewall-cmd --permanent --policy policy_int_to_ext --add-egress-zone external
firewall-cmd --permanent --policy policy_int_to_ext --set-priority 100
firewall-cmd --permanent --policy policy_int_to_ext --set-target ACCEPT
firewall-cmd --permanent --zone=external --add-masquerade
systemctl restart firewalld
firewall-cmd --info-policy policy_int_to_ext


Posted in Uncategorized | Tagged | 2 Comments

Note to self: Windows install

Install Chocolaty (Packagemanger for Windows) from

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString(''))

Install needed software

choco install chocolateyGUI 7zip cdburnerxp crystaldiskinfo firefox libreoffice-fresh putty sumatrapdf sysinternals vlc windirstat shutup10 teamviewer nextcloud-client

run oosu10 (Meta-r oosu10)

Posted in Uncategorized | Leave a comment

Happy Backup Day

Finally I can write my happy “backup day” blog post, after several years of problems with my own backup. Even when it is a couple of days to late 😉 The last weeks I create a new backup setup and rolled it out to all my “production” systems. Yeahh.

Here is my new / current setup:


During my search for a backup solution I stumbled upon the tool backupninja and really liked the idea. Don’t write your own script that calls DB backup and the filesystem backup, but instead drop a file inside /etc/backup.d/ and it will automatically does it for you and sends an email if there is a problem. Even when this is a very Debian centric project, there is a Fedora/EPEL package for it. But the EPEL Package is only on version 1.1.0, instead of the current 1.2.1, but a mock rebuild of the fedora package for RHEL7 fixed that without any major issues. 1.2.1 is important because now restic is supported. There is still some love needed, because there are now at least two trees that are drifting apart ;-(

restic vs borg

The part that took the longest was the decision restic vs. borg. I used duplicity before, but the necessity to make a full backup regularly was a big NO for me. Even when duplicity is the only backup – that I know off – that supports repairing of errors within the archive with the help of par2 files. But at the End the decision was borg vs. restic. After some considerations I decided to simply start with both and decide later 😉 Better two backups than no backups.

Android Backup = seedvault vs. TitaniumBackup

I’m using titanium Backup for Android for a very long time. It works good, it can be bought without a google account and I moved between multiple Devices by restoring a backup on the new device. But the need for a rooting solution makes it hard for devices other than my personal own ( dor my wife, friends etc.). I was surprised when I learned today the my LineageOS already have a backup solution build in. It was something that was mention in the release-notes of LineageOS 18.1: seedvault. I was shocked to find it already in current LineageOS 17.1. My highlight was the direct support for nextcloud as a backup target. I did not tested the restore yet, but that will be next 😉 So at the moment I also have 2 backup solution for my Google free Android phone.

Password and Bookmark sync and Backup

My last paint point when it comes to backup was backup of my Browser bookmark and Passwords including sync. Of course I could use Chromium or Firefox buildin sync mechanism. (I use both browser at the moment, but I still use primarily firefox. In part because I use the buildin password store. )

An article about bitwarden and podman was posted in the fedora magazin years ago and I was playing around since about a year ago. Because Chromium was forbidden to use the sync from Google Chrome soon (or already), the idea was regrowing in my head.

The nice thing about bitwarden is that it is OpenSource and has the possibility to run it self hosted, but it is quite a heavy setup with more than 10 containers running and it has to be real docker, not podman ;-(. But bitwarden_rs to the rescue. It is a rewrite in rust, that is using a sqlite database and can run without docker. Because it is a rewrite it does not support all options from the full setup, but for a small setup it is ideal. The migration is not yet done, but that will hopefully happen in the next couple of weeks.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Leave a comment

Centos and the end of a horrible year (Part II)

2020 will be for most people one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or better Covid-19. As I’m a member of a risk group, my social life is already for almost 9 month limited to telephone and video calls.

And on top of all that, in the last month of 2020 two of my favorite software projects changed directions and removed the main reasons why I’m using them. This blog post is about Centos. The Graylog Blog post was Part I.


I use Red Hat since 1999. First Red Hat 6, then I got my RHCE with Red Hat 6.2. Which was fun to explain when RHEL6 was released ;-). I moved to Fedora with the release 1 and started using it as my main desktop machine since then. I was the second RHCA in Europe and #8 worldwide. I was a professionel Red Hat trainer and examiner from 2000 to 2010. I owe my job in IT partly to Red Hat. I’m a Fedora ambassador and Fedora packager. But due to time restrains, both are dormant for some time now. I also was in the room, when for the first time, people from Red Hat were talking to the CentOS team at a Fosdem conference many years ago. I was sitting in the room at a Red Hat Summit when Ginni Rometty told Jim Whitehurst that “Red Hat will be a separate entity inside IBM“. Each time trust was the most important thing for me.

I really like Red Hat, even when I was never employed by them, only freelancing for them. I have an original red Fedora from the IPO of Red Hat stocks in 1999 and I learned to trust Red Hat since then. Red Hat said we only do Open Source, and when they bought Netscape in 2004, it took them a long time, but finally the Fedora Directory Server (now called 389 Directory Server) was released as an Open Source product. When Red Hat bought Qumranet, they released their KVM based Virualization Solution (now Ovirt). OK, they had to port it away from Windows and IIS first, but finally they released it. When Red Hat bought ansible and they also released the Ansible Tower, as they said they would.

The same trust was applied when Red Hat’s CTO Chris Wright said in an interview a year ago: ‘So, if you need a stable RHEL-like operating system, CentOS will still be there for you‘ and ‘In other words, nothing changes for current users of CentOS.

But in December of 2020 my trust in Red Hat and Centos (now part of Red Hat, now part of IBM) was deeply eroded (if not shuttered) when the removal of Centos 8 was announced. In the beginning of the year the webpage of Centos stated Centos 8 will be supported until 2028. But unfortunately in December 2020 it was changed to end of the year 2021 instead.

I use Centos Stream 8 on one of my machines and with that I can have a look at the future of RHEL 8 and therefore Centos 8. I really liked that, and saw huge potential for Centos. The look on RHEL 8.4 Beta would be boring, because I was already running Centos Stream 8. The same way in which a look at Fedora 33 gives you a glimps into RHEL 9, or more precise, at Fedora ENL (=Enterprise Linux Next).

I have a couple of internal machines running with Fedora, where I have no problems that I have to update all the time. As the machines are not so important.

But my personal main machine – running my mailserver, webserver etc. – I want to run on a stable platform, without the need to jump versions all the time. It should be based on RHEL, because I really like Red Hat, rpm and especially SELinux.

One of the questions on everybody’s mind is: “How much influence had IBM in this decision?”. I don’t know, but after almost 20 years of dealing with Red Hat, a broken promise occurring directly after the IBM / Red Hat takeover is – at least – suspicious.

Finally Centos will not support version 8 for more than 5 years (ok, maybe 6). That is OK, but not the promised 10 years. Because the updates for Red Hat are so hard that a re-install is needed every 10 years. This is good to get rid of all the old stuff, that was laying around and got rotten.

If you are updated from 7 to 8 already, than you will likely to update to 9 before 8 is EOL, but if you have updated already in the last 2 years from 6 to 8, than Centos is not a solution for you anymore.

What is the end effect of all this? I see three possible future scenarios, for people who want the promised 10 years:

1.) Red Hat re-thinks their decision and they stay to their word – continuing with Centos 8 as promised and switching to the new system with ‘Centos 9’. This way, they keep their word, because changing the game with a new version – is fair game and not a broken promise.

2.) Rocky Linux is available and running at the EOL time. Anyone who wants to run a stable RHEL-like distro can move to Rocky Linux 8 and stay there until RHEL 8 is EOLed.

3.) Oracle Linux. Yes, Oracle Linux. Yes, I have the “Unfakeable Linux” T-Shirt and wear it proud and often. Nevertheless I was testing OL7 when Centos 7 was taking a very long time to appear. Also they still offer a RHEL like free distribution with full 10 years of lifetime. Something that Centos did for a long time and will not offer any longer. Of course, I would prefer Rocky Linux, but Oracle Linux is already available and I’m sure a lot of Centos User who want long-time stability will move there. Great idea of Red Hat – pushing people to Oracle. Really a great move. This is what your breaking of trust has led to. Of course I don’t trust that Oracle Linux will continue to offer free OL forever, but I did trust in the past, that Red Hat will do what they announced. And what has this trust led me to?

P.S.: But at the end one a more positive note: Maybe the Centos Team has now more time to support an update from Centos Stream 8 to Centos Stream 9. One can only hope!

Posted in Enterprise Linux, Fedora | Tagged | 1 Comment

Graylog or the end of a horrible year (Part I)

2020 will be – for most people – one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or Covid-19. As I’m part of a risk group, my social life is for almost 9 month limited to telephone and video calls.

And on top of it all – two of my favorite software projects dramatically changed directions in December 2020 and removing the main reasons why I’m using them. This blog post is about Graylog. The Centos Blogpost comes later in Part II.


I’m using graylog for a very long time. Starting 2012 I began using it privately and urged my customers to use it as well. I know of at least 2 companies that started using it because of my endorsement and one of them switched to the enterprise version. I gave multiple talks about Structured Logfiles (for example @ Froscon 2013 – look at the Graylog 2 page ;-))
I finished the talk with the ‘catch’ that graylog was – at that time – one of the few Open Core Businesses that got a good split between the Open Source and Closed Source addon. In the last couple of years I did not give that much talks anymore, but still supported customers.

The big difference between other solutions like ELK, was that the Open Source Version of Graylog included a real integration into ADS. So you could decide based on LDAP Groups who could see which kind of messages. The Enterprise version has some very good features too, including Archiving and Auditing.

But with Graylog 4 they removed the possibility of using LDAP Groups, because they restructured LDAP Groups and now declared it an Enterprise feature called Teams.

This is – imho – a very bad business decision. I would like to explain why: one department of my customer has switched to Graylog about 2 years ago. It was free and more and more developers – after some growing pains – really embraced Graylog. After the implementation, it was used more and more and the enterprise version was bought because support was needed and the Archiving and Auditing etc. became more important. Another department in the same company looked also at graylog, but because now they have to buy the enterprise version immediately, they are looking more intensive at other options as well. Now there are also including ELK, Splunk and the integrated EFK Stack in OpenShift in their search. Because all of them are used at some department in the company as well. At the moment it looks that graylog will likely lose at the end and will not be the tool chosen.

To make matters worse – there was not a good communication of change: No warning in the release notes, hey this will go away in the open source version with release XY. No, the releas notes only read: ‘the … old Groups .. have been replaced by Teams in Graylog Enterprise‘. The only good information was in the bug tracker, with other users complaining. I would have hoped for something like that in an official release note: “For Open Source Users of graylog the LDAP and Active Directory group mapping have been removed.

If someone asked me for a Logging Solution in October 2020 my answer would have been Graylog – pure and simple. Now, I will lay out multiple solutions.

Posted in Fedora, logs | Tagged | Leave a comment

Hey, I just need a virtual Machine with …

If you just need a fast (manual) installation with a certain OS. Instead of downloading the ISO and than start the install. Just use virt-install that will do the download, startup etc. for you.

Yes, I know there are better way with images, but if you want an custom installation this is easiest.

virt-install --install centos8 --name  c8
virt-install --install debian10 --name d10
virt-install --install fedora33 --name fed33
virt-install --install centos7.0 --name c7

Yes, it is Centos7.0 for the current Centos7. Centos 7.8 and Centos7 does not work.
With Centos8 and centos-stream8 it uses a better name.

You get a complete list of supported OSs with osinfo-query os.

If you don’t want to use the default disk, memory or CPU you have to specify it.

Here my test for a Secure-boot TPM Install to test Clevis-TPM.

virt-install -n Secure-boot-test --install centos8 --memory=3072 --vcpu=2 --disk size=60,bus=scsi,sparse=true,discard=unmap --disk size=60,bus=scsi,sparse=true,discard=unmap -w network=default --controller scsi,model=virtio-scsi --boot uefi,loader=/usr/share/OVMF/OVMF_CODE.secboot.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/OVMF/OVMF_VARS.secboot.fd,loader_secure=yes --features smm.state=on --tpm model="tpm-tis",backend.type="emulator",backend.version="2.0"
Posted in Enterprise Linux, Fedora | Tagged | Leave a comment

Centos 7 to 8 Update


Disclaimer: This is not supported and it can breaks you computer or delete your data. You have been warned. I take no responsibility if it breaks. Double check that your backup is working.


My main server was updated from Centos 3 -> 4 -> 5 with the help of the undocumented upgrade option of Install CD. So I have a lot of experience with “unofficial” updating Centos. In between it was also moved from physical to a virtual machine. This did not work with update 6 -> 7. The project to port the official red hat tool to CEntOS did never took off.

At the end I had to reinstall my machine, but after more the 10 years of updates, it was time to cleanup, reinstall, configure and check everything. During the cleanup I found a couple of very surprising things that still hung around 🙂


Now with the release Cenos 8. I don’t want to do that again (the 10 years are not up yet) and I tried to run an upgrade from Centos 7 -> 8. It kind of works, but of course it is not for the faint hearted. Yes, I did it in a test environment first.
An update from within a running Centos 7 did not work, because even with the available dnf, the old rpm from C7 was not able run the update.

EPEL and others

If you are using EPEL or other repos this procedure could remove some or all of the packages from this repos. Especially epel is still quite empty at the moment, also other projects do not support Centos8 yet.


So you have to boot the Centos8 image and start a Centos8 rescue system.

You network configuration (network port name) could change as well, so be sure you can access the console (directly or via remote admin card)

You are asked if the file system should be mounted, please answer with yes.
The commands are quite simple:

rpm --root /mnt/sysimage/ -Uvh /mnt/install/repo/BaseOS/Packages/centos-release-*.x86_64.rpm
dnf --installroot=/mnt/sysimage/ distro-sync --allowerasing

It is possible that you have to remove some of the packages because it is missing the correct replace line in the spec. In my case I had to remove:

rpm --root /mnt/sysimage/ -e --nodeps systemvinit-tools python-inotify pycairo yum adwaita-qt5
dnf --installroot=/mnt/sysimage/ distro-sync --allowerasing
dnf --installroot=/mnt/sysimage/ install @base

If you use other groups like “Server with Gui” please install this group as well.

If you want to use a different language then English, please install the needed langpack like:

dnf --installroot=/mnt/sysimgae/ install glibc-langpack-de

If you prefer to use yum instead of dnf or if you had remove something that you need, you can do it now:

dnf --installroot=/mnt/sysimgae/  install yum


After that you can boot into your server again. It will take a longer time than usual, because a relabel will occur at the first boot. Be patient.

Cleanup the old repos:

rm -rf /var/cache/yum/ /var/cache/dnf/
dnf update --refresh

You can try to remove packages from EL7. But please be careful and check the removed packages. I had to install “@Base” and “@Server with Gui” again:

dnf repoquery --extras -q | xargs dnf remove

I urge you to run the following command and check the output:

dnf repoquery --duplicates 
dnf repoquery --unneeded 
dnf repoquery --extras 
dnf repoquery --unsatisfied

I clean out the rest of network-scripts as well and switch completely to NetworkManager:

dnf install NetworkManager-config-server
dnf remove network-scripts

I ran into a new intial-setup screen during every boot. To disable it I had to run:

systemctl disable intial-setup-reconfiguration.service

After that you have updated to Centos8.


I showed that it is possible to update from Centos7 to 8, but I’m not sure if I can trust this system after this upgrade. I will play around with test system at home and will write a new blog post, or update this post when I have decided for myself.

Posted in Enterprise Linux, Fedora, Linux | 2 Comments

Cool new tools

Just back from my vacation I learned two cool new tools on the first day. Of course both are in EPEL and Fedora.

  • The Silver Search: ag $STRING is an alternative to grep -ir $STING . But a lot of the hell faster, nicer output and multithreaded.
  • myrepos: From the developer of etckeeper, to pull, commit and doing other stuff with multiple repos at the same time. Simply got to each git repo run mr register and than to a mr pull to pull all git repos at the same time.



Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | 1 Comment

Moving my Server with less than a second downtime

As I’ve written in my previous post, I moved to a new Hetzner Box. This is how I setup my virtual Host and how I moved my server, including all virtual machines, to the new data center with a downtime of a second or less. This setup is only possible if hetzner allows you to move you network via datacenter.


  • Use host routing to avoid wasting IPs
  • install centos-release-qemu-ev to install kvm/qemu from SigVirt/OVirt/RHEV
  • With the newer kvm/qemu you can use virsh migrate --live --copy-storage-all

Don’t waste IP addresses

Because fixed IP addresses are expensive wasting two of the eight IPs to have a broadcast network is not an option.

Virtual Machine setup

We create virtual Interfaces vif0 to vif7 and configure the routing to which we connect the virtual machines on eth0. We let libvirt call a little script on startup of the virtual machines. To do this we add the configurations to the libvirt/quemu xml config of the VMs with virsh edit virt0:

<interface type='ethernet'>
      <mac address='XX:XX:XX:XX:XX:XX'/>
      <script path='/etc/libvirt/scripts/vif-route'/>
      <target dev='vif0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

which calls:


# $1 is interface

case "$1" in
      ip r a xx.xx.xx.x0 dev $1 &> /dev/null
      ip r a xx.xx.xx.x1 dev $1 &> /dev/null
      ip r a xx.xx.xx.x2 dev $1 &> /dev/null
      ip r a xx.xx.xx.x3 dev $1 &> /dev/null
      ip r a xx.xx.xx.x4 dev $1 &> /dev/null
      ip r a xx.xx.xx.x5 dev $1 &> /dev/null
      ip r a xx.xx.xx.x6 dev $1 &> /dev/null
      ip r a xx.xx.xx.x7 dev $1 &> /dev/null
echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp

exit 0

Setup inside the virtual Machine

Now we configure the network and routing for the virtual machines in /etc/sysconfig/network-scripts/ It should be noted, that the IP address is arbitrary. We need it becouse we cannot add a default route directly to a device, in this case eth0.




XX.XX.XX.YY/ZZ dev eth0
default via XX.XX.XX.YY dev eth0


Now we want to move the VM to our new server.

The old server, let’s call it Hetzner-Old, is a CentOS 7 with a bridged net/29 network and several virtual machines running on it (e.g. this Blog). The VMs are running on LVMs which themself are running on crypto devices. Each VM has its own LVM group.

The new server, Hetzner-New is also a CentOS 7 machine. The LVM devices have the same name and are of the same size or bigger. They have to have the same names. This is also a good opportunity to make an LVM device bigger without the need to shut down the according VM.

Now we add the repo for SIG Virtualisation on both machines.

yum install centos-release-qemu-ev

and update qemu/kvm etc

Next we open a tunnel between the two machines with

ssh -w 2:2 tun2-IP

and add ipaddesess on both sides (I will use ifconfig because othwise you have to use two commands)

ifconfig tun2 192.168.0.X/24

This way we have encrytion and don’t have to worry encryting it with TLS on qemu.

Server Diagramm

On Hetzner old we have to route through the tunnel tun2

ip r a net/29 via tun2-IP

Next we do

echo 0 | tee /proc/sys/net/ip4/conf/*/rp_filter

That way, when one of the network interfaces of a VM (vif+) gets removed, when the VM is moved, the according route is removed. Otherwise it would try to go through the default GW of Hetzner-New, wich is not allowed by Hetzner yet. Now, when the VM starts on Hetzner-New it has no interface to this IP-Address and it’s routed through the tunnel, through the GW of Hetzner-Old.

On Hetzner-New we add a new routing table with:

echo 201 vm-out >> /etc/iproute2/rt_tables

The number 201 is only internally used, therefore arbitrary.

Next we set a rule for every package where fwmark 1 is set use table vm-out:

ip rul a fwmark 1 table vm-out

Now we set a new default GATEWAY

ip r a default via IP-Hetzner-Old table vm-out

and set a firewall rule (vif+ are the interfaces for the VMs and iptables uses + instead of the commonly used *

iptables -t mangle -s net/29 -A PREROUTING -j MARK --set-mark 1 -i vif+

Now we are ready for moving the VMs from Hetzner-Old to Hetzner-New

virsh migrate --live VIRTUALMACHINE qemu+ssh://IP-HETZNER-Old/system --copy-storage-all --verbose --persistent

Finally we wait for Hetzner to swich the net/29 network over to Hetzner-New and throw away the previously set iptables rule. Since the new machine is in a different data center, Hetzner has to allow this moving. Because for this to work the net/29 network has to be in Hetzners core routers. With this command we wait for the first tcp package to arrive at Hetzner-New and flush our iptables rule.

tcpdump -i br0 net net/29 -s 1 && iptables -F PREROUTING -t mangle

Posted in Fedora | Tagged | Comments Off on Moving my Server with less than a second downtime

lmsensors for Monitoring a Hetzner EX41 with nct6775 on Centos7


I recently rented a new Hetzner Box to replace my old. I moved from EX40 to EX41 and saving even some money every month :-).

Every thing went smooth, but the sensors did not work. I had to use nct6775, but the module from Centos7 said “No Device”. ElRepo to the rescue. They have a nct6775 kmod available, but this module is not compatible with Centos7.4 :-(.

First I had to created a fixed package. I changed only the release-number and the kversion to  693.2.2 instead of 327. The fixed version is available at: I also open a ticket with elrepo: I don’t know how they will fix this. ElRepo uses kABI-tracking kmods and things like that should not happen. We will see.

With this fixed package installed I created the following configuration file (/etc/sensors.d/hetzner) with the help of sensors -u:

chip "nct6792-*"
 label in0 "2V"
 label in2 "3.3V1"
 label in3 "3.3V2"
 label in6 "1V1"
 label in7 "3.3V3"
 label in8 "3.3V4"
 label in9 "1V2"
 label in12 "1.1V"
 label fan1 "fan"

 set in0_min 2 * 0.80
 set in0_max 2 * 1.2
 # Can not set alarms
 #set in0_alarm: 0.000

 set temp1_max 40
 set temp1_max_hyst 38
 set temp2_max 40
 set temp2_max_hyst 38
 #set temp3_input 40
 # PECI Agent 0
 #set temp7_max 40

 ignore in1
 ignore in4
 ignore in5
 ignore in10
 ignore in11
 ignore in13
 ignore in14
 ignore fan2
 ignore temp4
 ignore temp5
 ignore temp6
 ignore temp8
 ignore temp9
 ignore temp10

 ignore intrusion0
 ignore intrusion1
 ignore beep_enable

I can not fix the alarm for 2V (in0), but the rest looks good. Don’t forget to set the setting with sensors -s.

Update 15.10.2017: The module kmod-nct6775 is now update in elrepo directly, so only the configuration file is needed.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Comments Off on lmsensors for Monitoring a Hetzner EX41 with nct6775 on Centos7