Not again Red Hat

Update 2024.05.26: I appears the elrepo has found a way to create kmod packages for RHEL9.4 and RHEL8.10. I tested mp3sas, ftsteutates and wireguard from the elrepo-testing repo and it works flawlessly.

In this video Jeff Geerling accounced that “Corporate Open Source is Dead”. He already dropped support from his really good ansible playbooks. This was because Red Hat only distributes its sources to customers. Another brick in this wall was announced today by the great ELrepo project.

In this blogpost it was announced that RHEL made some changes in the upcoming 8.10 and 9.4 releases of RHEL and this will break some of the kernel modules that were created by elrepo to allow running RHEL with older cards – that are not official supported anymore. The fun thing is not the whole driver that was deprecated, but only some of the supported pci-id where removed.

Especially for home lab users this created a big problem. aacraid, megaraid_sas, mlx4 and mpt3sas are drivers that are used in a lot of home labs everywhere.

Again the overall intention from Red Hat are not the problem. If Red Hat would break support of that in RHEL 10 there would be no problems. It would be interesting to know if this is a unexpected consequence of an patch or a targeted business decision. Yes I know why the support was dropped by Red Hat, but Red Hat is not only forgetting it roots, but again kicked the non-prod users in the curb. Just after they droped Centos and broke there promis there as well.

At least for my homelab this creates an extra work to do, because my RAID Controller is on the deprecated list. At least AlmaLinux has undo this patch and you don’t even need elrepo to support his older hardware.

I was planning to reinstall the host anyway. I only have to decide to select Fedora or AlmaLinux. The time to decide that is coming earlier the I hoped.

Posted in Enterprise Linux, Linux, Uncategorized | Tagged | 1 Comment

mostly unknown keepalived feature

There are lot of introduction to keepalived. Like Setting up a Linux cluster with Keepalived: Basic configuration | Enable Sysadmin ( or Keepalived and high availability: Advanced topics | Enable Sysadmin (

But I recently learned that keepalived has a cool feature that makes writing keepalived much easier (Thanks Spindy). Almost all documentation found on the net shows you that you need two different configuration files. But this is not the case. There is an extension that allows you to rollout the same configuration for all nodes.

When you compare this example from the first links. They use this two configurations:

server1# cat /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 51
        priority 255
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        virtual_ipaddress {
server2# cat /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 51
        priority 254
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        virtual_ipaddress {

When you look at it its almost the same file – only 2 lines are different, state and priority. Here comes the @format into play. You can rollout this file on both side and it will work the same way as above:

# cat /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
        @server1 state MASTER
        @server2 state BACKUP
        interface eth0
        virtual_router_id 51
        @server1 priority 255
        @server2 priority 254
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        virtual_ipaddress {

So you can configure lines that are only valid for certain hosts by adding a @HOSTNAME in front. More possibilities are explained in the man page keepalived.conf(5) in the section “Conditional configuration and configuration id”

Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | Leave a comment

podman-compose and systemd

I’m using more and more podman and especially podman-compose. podman-compose is not part of RHEL, but it is available in EPEL and it is in Fedora. Of course I run it as a non-root user. It really works great, but creating systemd unit files for podman-compose is ugly. I had it running for about a year, but I wanted to have a look for something better. This blog post talks about Fedora (tested with 39), RHEL8 and RHEL9. All have some smaller problems, but sometimes different ones.

I wanted to try Quadlet-podman for over a year. This week I had a closer look and found that it is more complicate than I thought. I really like the simple one-file solution of a compose file. I found podlet to migrate compose files to quadlet. (Use podlet musl, if you have problem with the glibc of the gnu version).

But at the end I really like to continue using the compose file that are provided by most of the tool that I use and I had only very small problems with podman-compose, all of them easy fixable. At the end I decided to use podman-compose systemd. There is not a lot of documentation, but I really liked it.

I had quite a lot of problems, but I will show you here how to fix them and how my setup is working now.

Setup as root

First things first. I run it always as non-root of course. If you do too, please do a “restorecon -R” to the homedir of the user that runs the container. audit2allow and the logs will not show the problem (you have to disable noaudit to see it – I fear), but it will interferes with the startup of your container.

You want to make sure the container user can run even when he is not loged in. So you have to enable linger with:

loginctl enable-linger USERNAME.

I enabled cgroupv2 on my RHEL8 and there is a bug that you have to fix. The Problem can be fixed by different solution, but I choose to change the file /etc/containers/containers.conf. Of course this is not needed on RHEL9 or Fedora.


To use podman-compose systemd you need a systemd template and I choose to set it up in /etc, because I have multiple user running multiple applications. (If the filename of the output of systemctl status on Fedora/RHEL9 looks strange to you. There is a link: /etc/xdg/systemd/user -> ../../systemd/user).

You can run the command podman-compose systemd -a create-unit as root. Or you can run this command as a normal user and paste the output to /etc/systemd/user/podman-compose@.service. But all platforms (podman-compose runs with version 1.0.6 on all) the template has an error that prohibits the successful startup of the containers. You have to add “–in-pod pod_%i” to the up command – thanks to RalfSchwiete. I also added the ExecStopPost line. Here my complete template:

# /etc/systemd/user/podman-compose@.service
Description=%i rootless pod (podman-compose)

ExecStartPre=-/usr/bin/podman-compose --in-pod pod_%i up --no-start
ExecStartPre=/usr/bin/podman pod start pod_%i
ExecStart=/usr/bin/podman-compose wait
ExecStop=/usr/bin/podman pod stop pod_%i
ExecStopPost=/usr/bin/podman pod rm pod_%i


As User

With the preparation as root finished, the setup as non-root is quit simple – Almost 😉 . First stop the container with “podman-compose down”. Then go to the directory with the podman-compose.yml (or docker-compose.yml if you use the old name) file and run “podman-compose systemd”. Be careful as this commands starts the containers again. I always stop the containers again with podman-compose down and start it up again with “systemctl –user enable –now ‘podman-compose@COMPOSE ‘ “. Otherwise you are not sure if the systemctl command is working.

But not on Fedora and RHEL9. Here I always got the error message “Failed to connect to bus: No medium found“. The solution was not to use “su – USERNAME” but instead

machinectl shell USERNAME@ 

With su the DBUS_SESSION_BUS_ADDRESS is missing on Fedora and RHEL9. This is a know issue, but RedHat states that “Using su or su – is not a currently supported mechanism of rootless podman.” I’m not sure it machinectl is supported or not, but I can tell you it is working. If you never heard of machinectl before or didn’t know that machinectl has a shell options – you are not alone. 🙂 The official way it to ssh as USERNAME into the maschine. (I prefer my way better :-))

Running but where are the log

If it is working you get this output at the end of the podman-compose systemd command like this:

you can use systemd commands like enable, start, stop, status, cat
all withoutsudo like this:

            systemctl --user enable --now 'podman-compose@COMPOSE'
            systemctl --user status 'podman-compose@COMPOSE'
            journalctl --user -xeu 'podman-compose@COMPOSE'

and for that to work outside a session
you might need to run the following command once

            sudo loginctl enable-linger 'USERNAME'

you can use podman commands like:

            podman pod ps
            podman pod stats 'pod_COMPOSE'
            podman pod logs --tail=10 -f 'pod_COMPOSE'

systemctl –user status ‘podman-compose@COMPOSE’ did work fine on RHEL8 and shows the output of the command. But on Fedora and RHEL9 it did not show anything. On all Version the command journalctl –user -xeu ‘podman-compose@COMPOSE’ never shows any output.

To fix this your non-root user has to become member of the systemd-journald group. But even then you have to use the right command on all Platforms. Not the one from the output above, but this instead:

journalctl  -xe --user-unit 'podman-compose@COMPOSE'

As you can see the podman-compose is quite nice, but there are a lot of stumbling block. After you know how to avoid them, it works quite well.

Posted in Enterprise Linux, Fedora, Uncategorized | Tagged | 1 Comment

Update Alma/RHEL with leapp and LUKS

I migrated most of my Centos 7 and 8 boxes to version 8 of RHEL or Alma Linux. (Exception is on a box with Arch Linux running on an old Atom CPU which did not support RHEL8).

But the update from RHEL/Alma 8 to 9 has one big problem. leapp does not support installation with luks disks via the actor inhibitwhenluks and most of my machines have encryption enabled.

I wanted to test what the problem is. So I installed a VM with Alma Linux 8 and luks. Delete the actor inhibitwhenluks and try to run leapp. I expected an unbootable system or some strange problem, but nothing happened. It simply worked.

The default VM setting is still BIOS on Fedora, so maybe the problem only exists with UEFI. So I deleted the VM and reinstalled it with UEFI and Virt-SCSI as disc controller ( don’t forget to add a SCSI Controller type Virt-SCSI). Installed again with root on luks, run leapp. Again no problem.

So from my short test with 2 virtual machines, it worked better then the warning suggested. I can only explain it with an abundance of cautiousness on the Red Hat side and a simple copy of the files from Alma side. This also explains an error that prohibits the usage on the Alma side, because there is error that the kernel is not from Red Hat and upgrade is not finished.

This is what I did on RHEL and Alma:

  • delete actor “prohibit luks”:
    rm -rf /usr/share/leapp-repository/repositories/system_upgrade/common/actors/inhibitwhenluks

For Alma you need to allow non RedHat Kernel to be installed:

  • remove abort in /usr/share/leapp-repository/repositories/system_upgrade/common/actors/kernel/checkinstalledkernels/libraries/
    sed -e 's:raise StopActorExecutionError:#raise StopActorExecutionError:g' /usr/share/leapp-repository/repositories/system_upgrade/common/actors/kernel/checkinstalledkernels/libraries/

That fixed this problem, at least for me. Of course the other problems from the “leapp preuptrade” have to be handled. I only tested it with a fresh install. I will update my homeserver in the next couple of days/weeks/month.

P.S.: Don’t forget to cleanup old gpg Key that are using SHA1 to fix the problem with: “Hash algorithm SHA1 not available.”. I delete all gpg-pubkey packages, because yum will reinstall the needed keys anyway.

Posted in Enterprise Linux | Tagged | 2 Comments

podman-compose in RHEL8

If you are – like me – running podman-compose from EPEL in RHEL8 and running into the error message “invalid config provided: extra host entries must be specified on the pod: network cannot be configured when it is shared with a pod” the problem is a incompatibility between the very old version of podman-compose in EPEL8 and podman 4.1.

An update to a new version of podman-compose is not possible because of a missing dependency of python-dotenv. But thanks to Gwyn Ciesla an update has been pushed to testing EPEL8 and an update of podman-compose could be soon available. Thanks a lot Gwyn!

Until than simply run “dnf module switch-to container-tools:4.0” and continue running until an update version of podman-compose is available.

Update: Thanks to Gwyn Ciesla the update is available and a “dnf module switch-to container-tools:rhel8″ will fix it (hopefully).

Posted in Enterprise Linux, Linux, Uncategorized | Tagged | 2 Comments

My podman journey so far

This blog post was rotting in my queue since November of 2021, but the info is still helpfull for someone, so I post it now.

I had a long time ago a look at docker when just version 1 was released, but I never really used it that much for my own production setups. I used it for playing around. I hated that is runs as root and even Docker itself had problem making it secure.

I really liked it when podman was released, I’m a fan of Dan Walsh from SELinux times and I really liked the idea of splitting it up and running it as non root (I know docker can do it now as well). But I did not have the time to play with it. But because my RSS Feeder (ttrss) does not support normal installation anymore I was forced to try it again. I started with bitwarden_rs and now ttrss.

Thinks I learned

I want to run my container as non-root of course and start it with systemd during boot time. The output of podman generate systemd is not really good, even with the parameters, because it does not add a User setting etc.. So I write my own unit file:

Description=Podman container-bitwarden_rw.service



ExecStartPre=-/usr/bin/podman rm -f bitwarden_rs
ExecStart=/usr/bin/podman run –userns=keep-id -e WEBSOCKET_ENABLED=true -e SIGNUPS_ALLOWED=false -e ROCKET_PORT=8080 –name bitwarden_rs -v XXX/:/data/:Z -p 8080:8080 -p 3012:3012 bitwardenrs/server:latest

ExecReload=-/usr/bin/podman stop bitwarden_rs
ExecReload=-/usr/bin/podman rm bitwarden_rs
ExecStop=-/usr/bin/podman stop bitwarden_rs



If you get the error message like /run/user/1000/libpod you need to run as root:

loginctl enable-linger ttrss


ttrss needes docker-compose, but podman-compose to the rescue. It’s only in epel yet, but I hope it will become part of the standard EL OS. I works great no problems. Even as non-root user. The startup script not really a problem, here my take on it:

Description=Podman container-ttrss.service


ExecStart=/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down
ExecStop=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down



If you want to start it automatically you need the user to start automatically. Same as before run this as root:

loginctl enable-linger ttrss

Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | Leave a comment

Firewalld Fedora 34 -> 35 Masquerade between Zones not working anymore

I updated my firewall from 34 to 35 and my firewall was not working anymore. There is a not good documented change with the release of firewalld 1.0 that hit me.

The fix is simple if you found it.

firewall-cmd --permanent --new-policy policy_int_to_ext
firewall-cmd --permanent --policy policy_int_to_ext --add-ingress-zone public
firewall-cmd --permanent --policy policy_int_to_ext --add-egress-zone external
firewall-cmd --permanent --policy policy_int_to_ext --set-priority 100
firewall-cmd --permanent --policy policy_int_to_ext --set-target ACCEPT
firewall-cmd --permanent --zone=external --add-masquerade
systemctl restart firewalld
firewall-cmd --info-policy policy_int_to_ext


Posted in Uncategorized | Tagged | 2 Comments

Note to self: Windows install

Install Chocolaty (Packagemanger for Windows) from

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString(''))

Install needed software

choco install chocolateyGUI 7zip cdburnerxp crystaldiskinfo firefox libreoffice-fresh putty sumatrapdf sysinternals vlc windirstat shutup10 teamviewer nextcloud-client

run oosu10 (Meta-r oosu10)

Posted in Uncategorized | Leave a comment

Happy Backup Day

Finally I can write my happy “backup day” blog post, after several years of problems with my own backup. Even when it is a couple of days to late 😉 The last weeks I create a new backup setup and rolled it out to all my “production” systems. Yeahh.

Here is my new / current setup:


During my search for a backup solution I stumbled upon the tool backupninja and really liked the idea. Don’t write your own script that calls DB backup and the filesystem backup, but instead drop a file inside /etc/backup.d/ and it will automatically does it for you and sends an email if there is a problem. Even when this is a very Debian centric project, there is a Fedora/EPEL package for it. But the EPEL Package is only on version 1.1.0, instead of the current 1.2.1, but a mock rebuild of the fedora package for RHEL7 fixed that without any major issues. 1.2.1 is important because now restic is supported. There is still some love needed, because there are now at least two trees that are drifting apart ;-(

restic vs borg

The part that took the longest was the decision restic vs. borg. I used duplicity before, but the necessity to make a full backup regularly was a big NO for me. Even when duplicity is the only backup – that I know off – that supports repairing of errors within the archive with the help of par2 files. But at the End the decision was borg vs. restic. After some considerations I decided to simply start with both and decide later 😉 Better two backups than no backups.

Android Backup = seedvault vs. TitaniumBackup

I’m using titanium Backup for Android for a very long time. It works good, it can be bought without a google account and I moved between multiple Devices by restoring a backup on the new device. But the need for a rooting solution makes it hard for devices other than my personal own ( dor my wife, friends etc.). I was surprised when I learned today the my LineageOS already have a backup solution build in. It was something that was mention in the release-notes of LineageOS 18.1: seedvault. I was shocked to find it already in current LineageOS 17.1. My highlight was the direct support for nextcloud as a backup target. I did not tested the restore yet, but that will be next 😉 So at the moment I also have 2 backup solution for my Google free Android phone.

Password and Bookmark sync and Backup

My last paint point when it comes to backup was backup of my Browser bookmark and Passwords including sync. Of course I could use Chromium or Firefox buildin sync mechanism. (I use both browser at the moment, but I still use primarily firefox. In part because I use the buildin password store. )

An article about bitwarden and podman was posted in the fedora magazin years ago and I was playing around since about a year ago. Because Chromium was forbidden to use the sync from Google Chrome soon (or already), the idea was regrowing in my head.

The nice thing about bitwarden is that it is OpenSource and has the possibility to run it self hosted, but it is quite a heavy setup with more than 10 containers running and it has to be real docker, not podman ;-(. But bitwarden_rs to the rescue. It is a rewrite in rust, that is using a sqlite database and can run without docker. Because it is a rewrite it does not support all options from the full setup, but for a small setup it is ideal. The migration is not yet done, but that will hopefully happen in the next couple of weeks.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Leave a comment

Centos and the end of a horrible year (Part II)

2020 will be for most people one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or better Covid-19. As I’m a member of a risk group, my social life is already for almost 9 month limited to telephone and video calls.

And on top of all that, in the last month of 2020 two of my favorite software projects changed directions and removed the main reasons why I’m using them. This blog post is about Centos. The Graylog Blog post was Part I.


I use Red Hat since 1999. First Red Hat 6, then I got my RHCE with Red Hat 6.2. Which was fun to explain when RHEL6 was released ;-). I moved to Fedora with the release 1 and started using it as my main desktop machine since then. I was the second RHCA in Europe and #8 worldwide. I was a professionel Red Hat trainer and examiner from 2000 to 2010. I owe my job in IT partly to Red Hat. I’m a Fedora ambassador and Fedora packager. But due to time restrains, both are dormant for some time now. I also was in the room, when for the first time, people from Red Hat were talking to the CentOS team at a Fosdem conference many years ago. I was sitting in the room at a Red Hat Summit when Ginni Rometty told Jim Whitehurst that “Red Hat will be a separate entity inside IBM“. Each time trust was the most important thing for me.

I really like Red Hat, even when I was never employed by them, only freelancing for them. I have an original red Fedora from the IPO of Red Hat stocks in 1999 and I learned to trust Red Hat since then. Red Hat said we only do Open Source, and when they bought Netscape in 2004, it took them a long time, but finally the Fedora Directory Server (now called 389 Directory Server) was released as an Open Source product. When Red Hat bought Qumranet, they released their KVM based Virualization Solution (now Ovirt). OK, they had to port it away from Windows and IIS first, but finally they released it. When Red Hat bought ansible and they also released the Ansible Tower, as they said they would.

The same trust was applied when Red Hat’s CTO Chris Wright said in an interview a year ago: ‘So, if you need a stable RHEL-like operating system, CentOS will still be there for you‘ and ‘In other words, nothing changes for current users of CentOS.

But in December of 2020 my trust in Red Hat and Centos (now part of Red Hat, now part of IBM) was deeply eroded (if not shuttered) when the removal of Centos 8 was announced. In the beginning of the year the webpage of Centos stated Centos 8 will be supported until 2028. But unfortunately in December 2020 it was changed to end of the year 2021 instead.

I use Centos Stream 8 on one of my machines and with that I can have a look at the future of RHEL 8 and therefore Centos 8. I really liked that, and saw huge potential for Centos. The look on RHEL 8.4 Beta would be boring, because I was already running Centos Stream 8. The same way in which a look at Fedora 33 gives you a glimps into RHEL 9, or more precise, at Fedora ENL (=Enterprise Linux Next).

I have a couple of internal machines running with Fedora, where I have no problems that I have to update all the time. As the machines are not so important.

But my personal main machine – running my mailserver, webserver etc. – I want to run on a stable platform, without the need to jump versions all the time. It should be based on RHEL, because I really like Red Hat, rpm and especially SELinux.

One of the questions on everybody’s mind is: “How much influence had IBM in this decision?”. I don’t know, but after almost 20 years of dealing with Red Hat, a broken promise occurring directly after the IBM / Red Hat takeover is – at least – suspicious.

Finally Centos will not support version 8 for more than 5 years (ok, maybe 6). That is OK, but not the promised 10 years. Because the updates for Red Hat are so hard that a re-install is needed every 10 years. This is good to get rid of all the old stuff, that was laying around and got rotten.

If you are updated from 7 to 8 already, than you will likely to update to 9 before 8 is EOL, but if you have updated already in the last 2 years from 6 to 8, than Centos is not a solution for you anymore.

What is the end effect of all this? I see three possible future scenarios, for people who want the promised 10 years:

1.) Red Hat re-thinks their decision and they stay to their word – continuing with Centos 8 as promised and switching to the new system with ‘Centos 9’. This way, they keep their word, because changing the game with a new version – is fair game and not a broken promise.

2.) Rocky Linux is available and running at the EOL time. Anyone who wants to run a stable RHEL-like distro can move to Rocky Linux 8 and stay there until RHEL 8 is EOLed.

3.) Oracle Linux. Yes, Oracle Linux. Yes, I have the “Unfakeable Linux” T-Shirt and wear it proud and often. Nevertheless I was testing OL7 when Centos 7 was taking a very long time to appear. Also they still offer a RHEL like free distribution with full 10 years of lifetime. Something that Centos did for a long time and will not offer any longer. Of course, I would prefer Rocky Linux, but Oracle Linux is already available and I’m sure a lot of Centos User who want long-time stability will move there. Great idea of Red Hat – pushing people to Oracle. Really a great move. This is what your breaking of trust has led to. Of course I don’t trust that Oracle Linux will continue to offer free OL forever, but I did trust in the past, that Red Hat will do what they announced. And what has this trust led me to?

P.S.: But at the end one a more positive note: Maybe the Centos Team has now more time to support an update from Centos Stream 8 to Centos Stream 9. One can only hope!

Posted in Enterprise Linux, Fedora | Tagged | 1 Comment