podman-compose in RHEL8

If you are – like me – running podman-compose from EPEL in RHEL8 and running into the error message “invalid config provided: extra host entries must be specified on the pod: network cannot be configured when it is shared with a pod” the problem is a incompatibility between the very old version of podman-compose in EPEL8 and podman 4.1.

An update to a new version of podman-compose is not possible because of a missing dependency of python-dotenv. But thanks to Gwyn Ciesla an update has been pushed to testing EPEL8 and an update of podman-compose could be soon available. Thanks a lot Gwyn!

Until than simply run “dnf module switch-to container-tools:4.0” and continue running until an update version of podman-compose is available.

Update: Thanks to Gwyn Ciesla the update is available and a “dnf module switch-to container-tools:rhel8″ will fix it (hopefully).

Posted in Enterprise Linux, Linux, Uncategorized | Tagged | 2 Comments

My podman journey so far

This blog post was rotting in my queue since November of 2021, but the info is still helpfull for someone, so I post it now.

I had a long time ago a look at docker when just version 1 was released, but I never really used it that much for my own production setups. I used it for playing around. I hated that is runs as root and even Docker itself had problem making it secure.

I really liked it when podman was released, I’m a fan of Dan Walsh from SELinux times and I really liked the idea of splitting it up and running it as non root (I know docker can do it now as well). But I did not have the time to play with it. But because my RSS Feeder (ttrss) does not support normal installation anymore I was forced to try it again. I started with bitwarden_rs and now ttrss.

Thinks I learned

I want to run my container as non-root of course and start it with systemd during boot time. The output of podman generate systemd is not really good, even with the parameters, because it does not add a User setting etc.. So I write my own unit file:

Description=Podman container-bitwarden_rw.service



ExecStartPre=-/usr/bin/podman rm -f bitwarden_rs
ExecStart=/usr/bin/podman run –userns=keep-id -e WEBSOCKET_ENABLED=true -e SIGNUPS_ALLOWED=false -e ROCKET_PORT=8080 –name bitwarden_rs -v XXX/:/data/:Z -p 8080:8080 -p 3012:3012 bitwardenrs/server:latest

ExecReload=-/usr/bin/podman stop bitwarden_rs
ExecReload=-/usr/bin/podman rm bitwarden_rs
ExecStop=-/usr/bin/podman stop bitwarden_rs


WantedBy=multi-user.target default.target

If you get the error message like /run/user/1000/libpod you need to run as root:

loginctl enable-linger ttrss


ttrss needes docker-compose, but podman-compose to the rescue. It’s only in epel yet, but I hope it will become part of the standard EL OS. I works great no problems. Even as non-root user. The startup script not really a problem, here my take on it:

Description=Podman container-ttrss.service


ExecStart=/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down
ExecStop=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down


WantedBy=multi-user.target default.target

If you want to start it automatically you need the user to start automatically. Same as before run this as root:

loginctl enable-linger ttrss

Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | Leave a comment

Firewalld Fedora 34 -> 35 Masquerade between Zones not working anymore

I updated my firewall from 34 to 35 and my firewall was not working anymore. There is a not good documented change with the release of firewalld 1.0 that hit me.

The fix is simple if you found it.

firewall-cmd --permanent --new-policy policy_int_to_ext
firewall-cmd --permanent --policy policy_int_to_ext --add-ingress-zone public
firewall-cmd --permanent --policy policy_int_to_ext --add-egress-zone external
firewall-cmd --permanent --policy policy_int_to_ext --set-priority 100
firewall-cmd --permanent --policy policy_int_to_ext --set-target ACCEPT
firewall-cmd --permanent --zone=external --add-masquerade
systemctl restart firewalld
firewall-cmd --info-policy policy_int_to_ext


Posted in Uncategorized | Tagged | 2 Comments

Note to self: Windows install

Install Chocolaty (Packagemanger for Windows) from https://chocolatey.org/install

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Install needed software

choco install chocolateyGUI 7zip cdburnerxp crystaldiskinfo firefox libreoffice-fresh putty sumatrapdf sysinternals vlc windirstat shutup10 teamviewer nextcloud-client

run oosu10 (Meta-r oosu10)

Posted in Uncategorized | Leave a comment

Happy Backup Day

Finally I can write my happy “backup day” blog post, after several years of problems with my own backup. Even when it is a couple of days to late ๐Ÿ˜‰ The last weeks I create a new backup setup and rolled it out to all my “production” systems. Yeahh.

Here is my new / current setup:


During my search for a backup solution I stumbled upon the tool backupninja and really liked the idea. Don’t write your own script that calls DB backup and the filesystem backup, but instead drop a file inside /etc/backup.d/ and it will automatically does it for you and sends an email if there is a problem. Even when this is a very Debian centric project, there is a Fedora/EPEL package for it. But the EPEL Package is only on version 1.1.0, instead of the current 1.2.1, but a mock rebuild of the fedora package for RHEL7 fixed that without any major issues. 1.2.1 is important because now restic is supported. There is still some love needed, because there are now at least two trees that are drifting apart ;-(

restic vs borg

The part that took the longest was the decision restic vs. borg. I used duplicity before, but the necessity to make a full backup regularly was a big NO for me. Even when duplicity is the only backup – that I know off – that supports repairing of errors within the archive with the help of par2 files. But at the End the decision was borg vs. restic. After some considerations I decided to simply start with both and decide later ๐Ÿ˜‰ Better two backups than no backups.

Android Backup = seedvault vs. TitaniumBackup

I’m using titanium Backup for Android for a very long time. It works good, it can be bought without a google account and I moved between multiple Devices by restoring a backup on the new device. But the need for a rooting solution makes it hard for devices other than my personal own ( dor my wife, friends etc.). I was surprised when I learned today the my LineageOS already have a backup solution build in. It was something that was mention in the release-notes of LineageOS 18.1: seedvault. I was shocked to find it already in current LineageOS 17.1. My highlight was the direct support for nextcloud as a backup target. I did not tested the restore yet, but that will be next ๐Ÿ˜‰ So at the moment I also have 2 backup solution for my Google free Android phone.

Password and Bookmark sync and Backup

My last paint point when it comes to backup was backup of my Browser bookmark and Passwords including sync. Of course I could use Chromium or Firefox buildin sync mechanism. (I use both browser at the moment, but I still use primarily firefox. In part because I use the buildin password store. )

An article about bitwarden and podman was posted in the fedora magazin years ago and I was playing around since about a year ago. Because Chromium was forbidden to use the sync from Google Chrome soon (or already), the idea was regrowing in my head.

The nice thing about bitwarden is that it is OpenSource and has the possibility to run it self hosted, but it is quite a heavy setup with more than 10 containers running and it has to be real docker, not podman ;-(. But bitwarden_rs to the rescue. It is a rewrite in rust, that is using a sqlite database and can run without docker. Because it is a rewrite it does not support all options from the full setup, but for a small setup it is ideal. The migration is not yet done, but that will hopefully happen in the next couple of weeks.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Leave a comment

Centos and the end of a horrible year (Part II)

2020 will be for most people one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or better Covid-19. As I’m a member of a risk group, my social life is already for almost 9 month limited to telephone and video calls.

And on top of all that, in the last month of 2020 two of my favorite software projects changed directions and removed the main reasons why I’m using them. This blog post is about Centos. The Graylog Blog post was Part I.


I use Red Hat since 1999. First Red Hat 6, then I got my RHCE with Red Hat 6.2. Which was fun to explain when RHEL6 was released ;-). I moved to Fedora with the release 1 and started using it as my main desktop machine since then. I was the second RHCA in Europe and #8 worldwide. I was a professionel Red Hat trainer and examiner from 2000 to 2010. I owe my job in IT partly to Red Hat. I’m a Fedora ambassador and Fedora packager. But due to time restrains, both are dormant for some time now. I also was in the room, when for the first time, people from Red Hat were talking to the CentOS team at a Fosdem conference many years ago. I was sitting in the room at a Red Hat Summit when Ginni Rometty told Jim Whitehurst that “Red Hat will be a separate entity inside IBM“. Each time trust was the most important thing for me.

I really like Red Hat, even when I was never employed by them, only freelancing for them. I have an original red Fedora from the IPO of Red Hat stocks in 1999 and I learned to trust Red Hat since then. Red Hat said we only do Open Source, and when they bought Netscape in 2004, it took them a long time, but finally the Fedora Directory Server (now called 389 Directory Server) was released as an Open Source product. When Red Hat bought Qumranet, they released their KVM based Virualization Solution (now Ovirt). OK, they had to port it away from Windows and IIS first, but finally they released it. When Red Hat bought ansible and they also released the Ansible Tower, as they said they would.

The same trust was applied when Red Hat’s CTO Chris Wright said in an interview a year ago: ‘So, if you need a stable RHEL-like operating system, CentOS will still be there for you‘ and ‘In other words, nothing changes for current users of CentOS.

But in December of 2020 my trust in Red Hat and Centos (now part of Red Hat, now part of IBM) was deeply eroded (if not shuttered) when the removal of Centos 8 was announced. In the beginning of the year the webpage of Centos stated Centos 8 will be supported until 2028. But unfortunately in December 2020 it was changed to end of the year 2021 instead.

I use Centos Stream 8 on one of my machines and with that I can have a look at the future of RHEL 8 and therefore Centos 8. I really liked that, and saw huge potential for Centos. The look on RHEL 8.4 Beta would be boring, because I was already running Centos Stream 8. The same way in which a look at Fedora 33 gives you a glimps into RHEL 9, or more precise, at Fedora ENL (=Enterprise Linux Next).

I have a couple of internal machines running with Fedora, where I have no problems that I have to update all the time. As the machines are not so important.

But my personal main machine – running my mailserver, webserver etc. – I want to run on a stable platform, without the need to jump versions all the time. It should be based on RHEL, because I really like Red Hat, rpm and especially SELinux.

One of the questions on everybody’s mind is: “How much influence had IBM in this decision?”. I don’t know, but after almost 20 years of dealing with Red Hat, a broken promise occurring directly after the IBM / Red Hat takeover is – at least – suspicious.

Finally Centos will not support version 8 for more than 5 years (ok, maybe 6). That is OK, but not the promised 10 years. Because the updates for Red Hat are so hard that a re-install is needed every 10 years. This is good to get rid of all the old stuff, that was laying around and got rotten.

If you are updated from 7 to 8 already, than you will likely to update to 9 before 8 is EOL, but if you have updated already in the last 2 years from 6 to 8, than Centos is not a solution for you anymore.

What is the end effect of all this? I see three possible future scenarios, for people who want the promised 10 years:

1.) Red Hat re-thinks their decision and they stay to their word – continuing with Centos 8 as promised and switching to the new system with ‘Centos 9’. This way, they keep their word, because changing the game with a new version – is fair game and not a broken promise.

2.) Rocky Linux is available and running at the EOL time. Anyone who wants to run a stable RHEL-like distro can move to Rocky Linux 8 and stay there until RHEL 8 is EOLed.

3.) Oracle Linux. Yes, Oracle Linux. Yes, I have the “Unfakeable Linux” T-Shirt and wear it proud and often. Nevertheless I was testing OL7 when Centos 7 was taking a very long time to appear. Also they still offer a RHEL like free distribution with full 10 years of lifetime. Something that Centos did for a long time and will not offer any longer. Of course, I would prefer Rocky Linux, but Oracle Linux is already available and I’m sure a lot of Centos User who want long-time stability will move there. Great idea of Red Hat – pushing people to Oracle. Really a great move. This is what your breaking of trust has led to. Of course I don’t trust that Oracle Linux will continue to offer free OL forever, but I did trust in the past, that Red Hat will do what they announced. And what has this trust led me to?

P.S.: But at the end one a more positive note: Maybe the Centos Team has now more time to support an update from Centos Stream 8 to Centos Stream 9. One can only hope!

Posted in Enterprise Linux, Fedora | Tagged | 1 Comment

Graylog or the end of a horrible year (Part I)

2020 will be – for most people – one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or Covid-19. As I’m part of a risk group, my social life is for almost 9 month limited to telephone and video calls.

And on top of it all – two of my favorite software projects dramatically changed directions in December 2020 and removing the main reasons why I’m using them. This blog post is about Graylog. The Centos Blogpost comes later in Part II.


I’m using graylog for a very long time. Starting 2012 I began using it privately and urged my customers to use it as well. I know of at least 2 companies that started using it because of my endorsement and one of them switched to the enterprise version. I gave multiple talks about Structured Logfiles (for example @ Froscon 2013 – look at the Graylog 2 page ;-))
I finished the talk with the ‘catch’ that graylog was – at that time – one of the few Open Core Businesses that got a good split between the Open Source and Closed Source addon. In the last couple of years I did not give that much talks anymore, but still supported customers.

The big difference between other solutions like ELK, was that the Open Source Version of Graylog included a real integration into ADS. So you could decide based on LDAP Groups who could see which kind of messages. The Enterprise version has some very good features too, including Archiving and Auditing.

But with Graylog 4 they removed the possibility of using LDAP Groups, because they restructured LDAP Groups and now declared it an Enterprise feature called Teams.

This is – imho – a very bad business decision. I would like to explain why: one department of my customer has switched to Graylog about 2 years ago. It was free and more and more developers – after some growing pains – really embraced Graylog. After the implementation, it was used more and more and the enterprise version was bought because support was needed and the Archiving and Auditing etc. became more important. Another department in the same company looked also at graylog, but because now they have to buy the enterprise version immediately, they are looking more intensive at other options as well. Now there are also including ELK, Splunk and the integrated EFK Stack in OpenShift in their search. Because all of them are used at some department in the company as well. At the moment it looks that graylog will likely lose at the end and will not be the tool chosen.

To make matters worse – there was not a good communication of change: No warning in the release notes, hey this will go away in the open source version with release XY. No, the releas notes only read: ‘the … old Groups .. have been replaced by Teams in Graylog Enterprise‘. The only good information was in the bug tracker, with other users complaining. I would have hoped for something like that in an official release note: “For Open Source Users of graylog the LDAP and Active Directory group mapping have been removed.

If someone asked me for a Logging Solution in October 2020 my answer would have been Graylog – pure and simple. Now, I will lay out multiple solutions.

Posted in Fedora, logs | Tagged | Leave a comment

Hey, I just need a virtual Machine with …

If you just need a fast (manual) installation with a certain OS. Instead of downloading the ISO and than start the install. Just use virt-install that will do the download, startup etc. for you.

Yes, I know there are better way with images, but if you want an custom installation this is easiest.

virt-install --install centos8 --name  c8
virt-install --install debian10 --name d10
virt-install --install fedora33 --name fed33
virt-install --install centos7.0 --name c7

Yes, it is Centos7.0 for the current Centos7. Centos 7.8 and Centos7 does not work.
With Centos8 and centos-stream8 it uses a better name.

You get a complete list of supported OSs with osinfo-query os.

If you don’t want to use the default disk, memory or CPU you have to specify it.

Here my test for a Secure-boot TPM Install to test Clevis-TPM.

virt-install -n Secure-boot-test --install centos8 --memory=3072 --vcpu=2 --disk size=60,bus=scsi,sparse=true,discard=unmap --disk size=60,bus=scsi,sparse=true,discard=unmap -w network=default --controller scsi,model=virtio-scsi --boot uefi,loader=/usr/share/OVMF/OVMF_CODE.secboot.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/OVMF/OVMF_VARS.secboot.fd,loader_secure=yes --features smm.state=on --tpm model="tpm-tis",backend.type="emulator",backend.version="2.0"
Posted in Enterprise Linux, Fedora | Tagged | Leave a comment

Centos 7 to 8 Update


Disclaimer: This is not supported and it can breaks you computer or delete your data. You have been warned. I take no responsibility if it breaks. Double check that your backup is working.


My main server was updated from Centos 3 -> 4 -> 5 with the help of the undocumented upgrade option of Install CD. So I have a lot of experience with “unofficial” updating Centos. In between it was also moved from physical to a virtual machine. This did not work with update 6 -> 7. The project to port the official red hat tool to CEntOS did never took off.

At the end I had to reinstall my machine, but after more the 10 years of updates, it was time to cleanup, reinstall, configure and check everything. During the cleanup I found a couple of very surprising things that still hung around ๐Ÿ™‚


Now with the release Cenos 8. I don’t want to do that again (the 10 years are not up yet) and I tried to run an upgrade from Centos 7 -> 8. It kind of works, but of course it is not for the faint hearted. Yes, I did it in a test environment first.
An update from within a running Centos 7 did not work, because even with the available dnf, the old rpm from C7 was not able run the update.

EPEL and others

If you are using EPEL or other repos this procedure could remove some or all of the packages from this repos. Especially epel is still quite empty at the moment, also other projects do not support Centos8 yet.


So you have to boot the Centos8 image and start a Centos8 rescue system.

You network configuration (network port name) could change as well, so be sure you can access the console (directly or via remote admin card)

You are asked if the file system should be mounted, please answer with yes.
The commands are quite simple:

rpm --root /mnt/sysimage/ -Uvh /mnt/install/repo/BaseOS/Packages/centos-release-*.x86_64.rpm
dnf --installroot=/mnt/sysimage/ distro-sync --allowerasing

It is possible that you have to remove some of the packages because it is missing the correct replace line in the spec. In my case I had to remove:

rpm --root /mnt/sysimage/ -e --nodeps systemvinit-tools python-inotify pycairo yum adwaita-qt5
dnf --installroot=/mnt/sysimage/ distro-sync --allowerasing
dnf --installroot=/mnt/sysimage/ install @base

If you use other groups like “Server with Gui” please install this group as well.

If you want to use a different language then English, please install the needed langpack like:

dnf --installroot=/mnt/sysimgae/ install glibc-langpack-de

If you prefer to use yum instead of dnf or if you had remove something that you need, you can do it now:

dnf --installroot=/mnt/sysimgae/  install yum


After that you can boot into your server again. It will take a longer time than usual, because a relabel will occur at the first boot. Be patient.

Cleanup the old repos:

rm -rf /var/cache/yum/ /var/cache/dnf/
dnf update --refresh

You can try to remove packages from EL7. But please be careful and check the removed packages. I had to install “@Base” and “@Server with Gui” again:

dnf repoquery --extras -q | xargs dnf remove

I urge you to run the following command and check the output:

dnf repoquery --duplicatesย 
dnf repoquery --unneeded 
dnf repoquery --extras 
dnf repoquery --unsatisfied

I clean out the rest of network-scripts as well and switch completely to NetworkManager:

dnf install NetworkManager-config-server
dnf remove network-scripts

I ran into a new intial-setup screen during every boot. To disable it I had to run:

systemctl disable intial-setup-reconfiguration.service

After that you have updated to Centos8.


I showed that it is possible to update from Centos7 to 8, but I’m not sure if I can trust this system after this upgrade. I will play around with test system at home and will write a new blog post, or update this post when I have decided for myself.

Posted in Enterprise Linux, Fedora, Linux | 2 Comments

Cool new tools

Just back from my vacation I learned two cool new tools on the first day. Of course both are in EPEL and Fedora.

  • The Silver Search: ag $STRING is an alternative to grep -ir $STING . But a lot of the hell faster, nicer output and multithreaded. https://geoff.greer.fm/ag/
  • myrepos: From the developer of etckeeper, to pull, commit and doing other stuff with multiple repos at the same time. Simply got to each git repo run mr register and than to a mr pull to pull all git repos at the same time. https://myrepos.branchable.com/



Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | 1 Comment