podman-compose and systemd

I’m using more and more podman and especially podman-compose. podman-compose is not part of RHEL, but it is available in EPEL and it is in Fedora. Of course I run it as a non-root user. It really works great, but creating systemd unit files for podman-compose is ugly. I had it running for about a year, but I wanted to have a look for something better. This blog post talks about Fedora (tested with 39), RHEL8 and RHEL9. All have some smaller problems, but sometimes different ones.

I wanted to try Quadlet-podman for over a year. This week I had a closer look and found that it is more complicate than I thought. I really like the simple one-file solution of a compose file. I found podlet to migrate compose files to quadlet. (Use podlet musl, if you have problem with the glibc of the gnu version).

But at the end I really like to continue using the compose file that are provided by most of the tool that I use and I had only very small problems with podman-compose, all of them easy fixable. At the end I decided to use podman-compose systemd. There is not a lot of documentation, but I really liked it.

I had quite a lot of problems, but I will show you here how to fix them and how my setup is working now.

Setup as root

First things first. I run it always as non-root of course. If you do too, please do a “restorecon -R” to the homedir of the user that runs the container. audit2allow and the logs will not show the problem (you have to disable noaudit to see it – I fear), but it will interferes with the startup of your container.

You want to make sure the container user can run even when he is not loged in. So you have to enable linger with:

loginctl enable-linger USERNAME.

I enabled cgroupv2 on my RHEL8 and there is a bug that you have to fix. The Problem can be fixed by different solution, but I choose to change the file /etc/containers/containers.conf. Of course this is not needed on RHEL9 or Fedora.

#/etc/containers/containers.conf
[containers]
pids_limit=0

To use podman-compose systemd you need a systemd template and I choose to set it up in /etc, because I have multiple user running multiple applications. (If the filename of the output of systemctl status on Fedora/RHEL9 looks strange to you. There is a link: /etc/xdg/systemd/user -> ../../systemd/user).

You can run the command podman-compose systemd -a create-unit as root. Or you can run this command as a normal user and paste the output to /etc/systemd/user/podman-compose@.service. But all platforms (podman-compose runs with version 1.0.6 on all) the template has an error that prohibits the successful startup of the containers. You have to add “–in-pod pod_%i” to the up command – thanks to RalfSchwiete. I also added the ExecStopPost line. Here my complete template:

# /etc/systemd/user/podman-compose@.service
[Unit]
Description=%i rootless pod (podman-compose)

[Service]
Type=simple
EnvironmentFile=%h/.config/containers/compose/projects/%i.env
ExecStartPre=-/usr/bin/podman-compose --in-pod pod_%i up --no-start
ExecStartPre=/usr/bin/podman pod start pod_%i
ExecStart=/usr/bin/podman-compose wait
ExecStop=/usr/bin/podman pod stop pod_%i
ExecStopPost=/usr/bin/podman pod rm pod_%i

[Install]
WantedBy=default.target

As User

With the preparation as root finished, the setup as non-root is quit simple – Almost 😉 . First stop the container with “podman-compose down”. Then go to the directory with the podman-compose.yml (or docker-compose.yml if you use the old name) file and run “podman-compose systemd”. Be careful as this commands starts the containers again. I always stop the containers again with podman-compose down and start it up again with “systemctl –user enable –now ‘podman-compose@COMPOSE ‘ “. Otherwise you are not sure if the systemctl command is working.

But not on Fedora and RHEL9. Here I always got the error message “Failed to connect to bus: No medium found“. The solution was not to use “su – USERNAME” but instead

machinectl shell USERNAME@ 

With su the DBUS_SESSION_BUS_ADDRESS is missing on Fedora and RHEL9. This is a know issue, but RedHat states that “Using su or su – is not a currently supported mechanism of rootless podman.” I’m not sure it machinectl is supported or not, but I can tell you it is working. If you never heard of machinectl before or didn’t know that machinectl has a shell options – you are not alone. 🙂 The official way it to ssh as USERNAME into the maschine. (I prefer my way better :-))

Running but where are the log

If it is working you get this output at the end of the podman-compose systemd command like this:

you can use systemd commands like enable, start, stop, status, cat
all withoutsudo like this:

            systemctl --user enable --now 'podman-compose@COMPOSE'
            systemctl --user status 'podman-compose@COMPOSE'
            journalctl --user -xeu 'podman-compose@COMPOSE'

and for that to work outside a session
you might need to run the following command once

            sudo loginctl enable-linger 'USERNAME'

you can use podman commands like:

            podman pod ps
            podman pod stats 'pod_COMPOSE'
            podman pod logs --tail=10 -f 'pod_COMPOSE'

systemctl –user status ‘podman-compose@COMPOSE’ did work fine on RHEL8 and shows the output of the command. But on Fedora and RHEL9 it did not show anything. On all Version the command journalctl –user -xeu ‘podman-compose@COMPOSE’ never shows any output.

To fix this your non-root user has to become member of the systemd-journald group. But even then you have to use the right command on all Platforms. Not the one from the output above, but this instead:

journalctl  -xe --user-unit 'podman-compose@COMPOSE'

As you can see the podman-compose is quite nice, but there are a lot of stumbling block. After you know how to avoid them, it works quite well.

Posted in Enterprise Linux, Fedora, Uncategorized | Tagged | Leave a comment

Update Alma/RHEL with leapp and LUKS

I migrated most of my Centos 7 and 8 boxes to version 8 of RHEL or Alma Linux. (Exception is on a box with Arch Linux running on an old Atom CPU which did not support RHEL8).

But the update from RHEL/Alma 8 to 9 has one big problem. leapp does not support installation with luks disks via the actor inhibitwhenluks and most of my machines have encryption enabled.

I wanted to test what the problem is. So I installed a VM with Alma Linux 8 and luks. Delete the actor inhibitwhenluks and try to run leapp. I expected an unbootable system or some strange problem, but nothing happened. It simply worked.

The default VM setting is still BIOS on Fedora, so maybe the problem only exists with UEFI. So I deleted the VM and reinstalled it with UEFI and Virt-SCSI as disc controller ( don’t forget to add a SCSI Controller type Virt-SCSI). Installed again with root on luks, run leapp. Again no problem.

So from my short test with 2 virtual machines, it worked better then the warning suggested. I can only explain it with an abundance of cautiousness on the Red Hat side and a simple copy of the files from Alma side. This also explains an error that prohibits the usage on the Alma side, because there is error that the kernel is not from Red Hat and upgrade is not finished.

This is what I did on RHEL and Alma:

  • delete actor “prohibit luks”:
    rm -rf /usr/share/leapp-repository/repositories/system_upgrade/common/actors/inhibitwhenluks

For Alma you need to allow non RedHat Kernel to be installed:

  • remove abort in /usr/share/leapp-repository/repositories/system_upgrade/common/actors/kernel/checkinstalledkernels/libraries/checkinstalledkernels.py
    sed -e 's:raise StopActorExecutionError:#raise StopActorExecutionError:g' /usr/share/leapp-repository/repositories/system_upgrade/common/actors/kernel/checkinstalledkernels/libraries/checkinstalledkernels.py

That fixed this problem, at least for me. Of course the other problems from the “leapp preuptrade” have to be handled. I only tested it with a fresh install. I will update my homeserver in the next couple of days/weeks/month.

P.S.: Don’t forget to cleanup old gpg Key that are using SHA1 to fix the problem with: “Hash algorithm SHA1 not available.”. I delete all gpg-pubkey packages, because yum will reinstall the needed keys anyway.

Posted in Enterprise Linux | Tagged | Leave a comment

podman-compose in RHEL8

If you are – like me – running podman-compose from EPEL in RHEL8 and running into the error message “invalid config provided: extra host entries must be specified on the pod: network cannot be configured when it is shared with a pod” the problem is a incompatibility between the very old version of podman-compose in EPEL8 and podman 4.1.

An update to a new version of podman-compose is not possible because of a missing dependency of python-dotenv. But thanks to Gwyn Ciesla an update has been pushed to testing EPEL8 and an update of podman-compose could be soon available. Thanks a lot Gwyn!

Until than simply run “dnf module switch-to container-tools:4.0” and continue running until an update version of podman-compose is available.

Update: Thanks to Gwyn Ciesla the update is available and a “dnf module switch-to container-tools:rhel8″ will fix it (hopefully).

Posted in Enterprise Linux, Linux, Uncategorized | Tagged | 2 Comments

My podman journey so far

This blog post was rotting in my queue since November of 2021, but the info is still helpfull for someone, so I post it now.

I had a long time ago a look at docker when just version 1 was released, but I never really used it that much for my own production setups. I used it for playing around. I hated that is runs as root and even Docker itself had problem making it secure.

I really liked it when podman was released, I’m a fan of Dan Walsh from SELinux times and I really liked the idea of splitting it up and running it as non root (I know docker can do it now as well). But I did not have the time to play with it. But because my RSS Feeder (ttrss) does not support normal installation anymore I was forced to try it again. I started with bitwarden_rs and now ttrss.

Thinks I learned

I want to run my container as non-root of course and start it with systemd during boot time. The output of podman generate systemd is not really good, even with the parameters, because it does not add a User setting etc.. So I write my own unit file:

[Unit]
Description=Podman container-bitwarden_rw.service
Wants=network.target
After=network-online.target

[Service]
User=bitwarden_rs
Group=bitwarden_rs

Type=simple
TimeoutStartSec=5m

ExecStartPre=-/usr/bin/podman rm -f bitwarden_rs
ExecStart=/usr/bin/podman run –userns=keep-id -e WEBSOCKET_ENABLED=true -e SIGNUPS_ALLOWED=false -e ROCKET_PORT=8080 –name bitwarden_rs -v XXX/:/data/:Z -p 8080:8080 -p 3012:3012 bitwardenrs/server:latest

ExecReload=-/usr/bin/podman stop bitwarden_rs
ExecReload=-/usr/bin/podman rm bitwarden_rs
ExecStop=-/usr/bin/podman stop bitwarden_rs

Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target default.target

If you get the error message like /run/user/1000/libpod you need to run as root:

loginctl enable-linger ttrss

podman-compose

ttrss needes docker-compose, but podman-compose to the rescue. It’s only in epel yet, but I hope it will become part of the standard EL OS. I works great no problems. Even as non-root user. The startup script not really a problem, here my take on it:

[Unit]
Description=Podman container-ttrss.service
Wants=network.target
After=network-online.target

[Service]
User=tinytinyrss
Group=tinytinyrss

Type=simple
TimeoutStartSec=5m
ExecStart=/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml up
ExecReload=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down
ExecStop=-/usr/bin/podman-compose -f XXX/ttrss-docker/docker-compose.yml down

Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target default.target

If you want to start it automatically you need the user to start automatically. Same as before run this as root:

loginctl enable-linger ttrss

Posted in Enterprise Linux, Fedora, Linux, Uncategorized | Tagged | Leave a comment

Firewalld Fedora 34 -> 35 Masquerade between Zones not working anymore

I updated my firewall from 34 to 35 and my firewall was not working anymore. There is a not good documented change with the release of firewalld 1.0 that hit me.

The fix is simple if you found it.

firewall-cmd --permanent --new-policy policy_int_to_ext
firewall-cmd --permanent --policy policy_int_to_ext --add-ingress-zone public
firewall-cmd --permanent --policy policy_int_to_ext --add-egress-zone external
firewall-cmd --permanent --policy policy_int_to_ext --set-priority 100
firewall-cmd --permanent --policy policy_int_to_ext --set-target ACCEPT
firewall-cmd --permanent --zone=external --add-masquerade
systemctl restart firewalld
firewall-cmd --info-policy policy_int_to_ext

Source

Posted in Uncategorized | Tagged | 2 Comments

Note to self: Windows install

Install Chocolaty (Packagemanger for Windows) from https://chocolatey.org/install

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Install needed software

choco install chocolateyGUI 7zip cdburnerxp crystaldiskinfo firefox libreoffice-fresh putty sumatrapdf sysinternals vlc windirstat shutup10 teamviewer nextcloud-client

run oosu10 (Meta-r oosu10)

Posted in Uncategorized | Leave a comment

Happy Backup Day

Finally I can write my happy “backup day” blog post, after several years of problems with my own backup. Even when it is a couple of days to late 😉 The last weeks I create a new backup setup and rolled it out to all my “production” systems. Yeahh.

Here is my new / current setup:

Backupninja

During my search for a backup solution I stumbled upon the tool backupninja and really liked the idea. Don’t write your own script that calls DB backup and the filesystem backup, but instead drop a file inside /etc/backup.d/ and it will automatically does it for you and sends an email if there is a problem. Even when this is a very Debian centric project, there is a Fedora/EPEL package for it. But the EPEL Package is only on version 1.1.0, instead of the current 1.2.1, but a mock rebuild of the fedora package for RHEL7 fixed that without any major issues. 1.2.1 is important because now restic is supported. There is still some love needed, because there are now at least two trees that are drifting apart ;-(

restic vs borg

The part that took the longest was the decision restic vs. borg. I used duplicity before, but the necessity to make a full backup regularly was a big NO for me. Even when duplicity is the only backup – that I know off – that supports repairing of errors within the archive with the help of par2 files. But at the End the decision was borg vs. restic. After some considerations I decided to simply start with both and decide later 😉 Better two backups than no backups.

Android Backup = seedvault vs. TitaniumBackup

I’m using titanium Backup for Android for a very long time. It works good, it can be bought without a google account and I moved between multiple Devices by restoring a backup on the new device. But the need for a rooting solution makes it hard for devices other than my personal own ( dor my wife, friends etc.). I was surprised when I learned today the my LineageOS already have a backup solution build in. It was something that was mention in the release-notes of LineageOS 18.1: seedvault. I was shocked to find it already in current LineageOS 17.1. My highlight was the direct support for nextcloud as a backup target. I did not tested the restore yet, but that will be next 😉 So at the moment I also have 2 backup solution for my Google free Android phone.

Password and Bookmark sync and Backup

My last paint point when it comes to backup was backup of my Browser bookmark and Passwords including sync. Of course I could use Chromium or Firefox buildin sync mechanism. (I use both browser at the moment, but I still use primarily firefox. In part because I use the buildin password store. )

An article about bitwarden and podman was posted in the fedora magazin years ago and I was playing around since about a year ago. Because Chromium was forbidden to use the sync from Google Chrome soon (or already), the idea was regrowing in my head.

The nice thing about bitwarden is that it is OpenSource and has the possibility to run it self hosted, but it is quite a heavy setup with more than 10 containers running and it has to be real docker, not podman ;-(. But bitwarden_rs to the rescue. It is a rewrite in rust, that is using a sqlite database and can run without docker. Because it is a rewrite it does not support all options from the full setup, but for a small setup it is ideal. The migration is not yet done, but that will hopefully happen in the next couple of weeks.

Posted in Enterprise Linux, Fedora, Linux | Tagged | Leave a comment

Centos and the end of a horrible year (Part II)

2020 will be for most people one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or better Covid-19. As I’m a member of a risk group, my social life is already for almost 9 month limited to telephone and video calls.

And on top of all that, in the last month of 2020 two of my favorite software projects changed directions and removed the main reasons why I’m using them. This blog post is about Centos. The Graylog Blog post was Part I.

Centos

I use Red Hat since 1999. First Red Hat 6, then I got my RHCE with Red Hat 6.2. Which was fun to explain when RHEL6 was released ;-). I moved to Fedora with the release 1 and started using it as my main desktop machine since then. I was the second RHCA in Europe and #8 worldwide. I was a professionel Red Hat trainer and examiner from 2000 to 2010. I owe my job in IT partly to Red Hat. I’m a Fedora ambassador and Fedora packager. But due to time restrains, both are dormant for some time now. I also was in the room, when for the first time, people from Red Hat were talking to the CentOS team at a Fosdem conference many years ago. I was sitting in the room at a Red Hat Summit when Ginni Rometty told Jim Whitehurst that “Red Hat will be a separate entity inside IBM“. Each time trust was the most important thing for me.

I really like Red Hat, even when I was never employed by them, only freelancing for them. I have an original red Fedora from the IPO of Red Hat stocks in 1999 and I learned to trust Red Hat since then. Red Hat said we only do Open Source, and when they bought Netscape in 2004, it took them a long time, but finally the Fedora Directory Server (now called 389 Directory Server) was released as an Open Source product. When Red Hat bought Qumranet, they released their KVM based Virualization Solution (now Ovirt). OK, they had to port it away from Windows and IIS first, but finally they released it. When Red Hat bought ansible and they also released the Ansible Tower, as they said they would.

The same trust was applied when Red Hat’s CTO Chris Wright said in an interview a year ago: ‘So, if you need a stable RHEL-like operating system, CentOS will still be there for you‘ and ‘In other words, nothing changes for current users of CentOS.

But in December of 2020 my trust in Red Hat and Centos (now part of Red Hat, now part of IBM) was deeply eroded (if not shuttered) when the removal of Centos 8 was announced. In the beginning of the year the webpage of Centos stated Centos 8 will be supported until 2028. But unfortunately in December 2020 it was changed to end of the year 2021 instead.

I use Centos Stream 8 on one of my machines and with that I can have a look at the future of RHEL 8 and therefore Centos 8. I really liked that, and saw huge potential for Centos. The look on RHEL 8.4 Beta would be boring, because I was already running Centos Stream 8. The same way in which a look at Fedora 33 gives you a glimps into RHEL 9, or more precise, at Fedora ENL (=Enterprise Linux Next).

I have a couple of internal machines running with Fedora, where I have no problems that I have to update all the time. As the machines are not so important.

But my personal main machine – running my mailserver, webserver etc. – I want to run on a stable platform, without the need to jump versions all the time. It should be based on RHEL, because I really like Red Hat, rpm and especially SELinux.

One of the questions on everybody’s mind is: “How much influence had IBM in this decision?”. I don’t know, but after almost 20 years of dealing with Red Hat, a broken promise occurring directly after the IBM / Red Hat takeover is – at least – suspicious.

Finally Centos will not support version 8 for more than 5 years (ok, maybe 6). That is OK, but not the promised 10 years. Because the updates for Red Hat are so hard that a re-install is needed every 10 years. This is good to get rid of all the old stuff, that was laying around and got rotten.

If you are updated from 7 to 8 already, than you will likely to update to 9 before 8 is EOL, but if you have updated already in the last 2 years from 6 to 8, than Centos is not a solution for you anymore.

What is the end effect of all this? I see three possible future scenarios, for people who want the promised 10 years:

1.) Red Hat re-thinks their decision and they stay to their word – continuing with Centos 8 as promised and switching to the new system with ‘Centos 9’. This way, they keep their word, because changing the game with a new version – is fair game and not a broken promise.

2.) Rocky Linux is available and running at the EOL time. Anyone who wants to run a stable RHEL-like distro can move to Rocky Linux 8 and stay there until RHEL 8 is EOLed.

3.) Oracle Linux. Yes, Oracle Linux. Yes, I have the “Unfakeable Linux” T-Shirt and wear it proud and often. Nevertheless I was testing OL7 when Centos 7 was taking a very long time to appear. Also they still offer a RHEL like free distribution with full 10 years of lifetime. Something that Centos did for a long time and will not offer any longer. Of course, I would prefer Rocky Linux, but Oracle Linux is already available and I’m sure a lot of Centos User who want long-time stability will move there. Great idea of Red Hat – pushing people to Oracle. Really a great move. This is what your breaking of trust has led to. Of course I don’t trust that Oracle Linux will continue to offer free OL forever, but I did trust in the past, that Red Hat will do what they announced. And what has this trust led me to?

P.S.: But at the end one a more positive note: Maybe the Centos Team has now more time to support an update from Centos Stream 8 to Centos Stream 9. One can only hope!

Posted in Enterprise Linux, Fedora | Tagged | 1 Comment

Graylog or the end of a horrible year (Part I)

2020 will be – for most people – one of the worst years in their living memory. The world is under lockdown because of SARS-Corona Virus 2 or Covid-19. As I’m part of a risk group, my social life is for almost 9 month limited to telephone and video calls.

And on top of it all – two of my favorite software projects dramatically changed directions in December 2020 and removing the main reasons why I’m using them. This blog post is about Graylog. The Centos Blogpost comes later in Part II.

Graylog

I’m using graylog for a very long time. Starting 2012 I began using it privately and urged my customers to use it as well. I know of at least 2 companies that started using it because of my endorsement and one of them switched to the enterprise version. I gave multiple talks about Structured Logfiles (for example @ Froscon 2013 – look at the Graylog 2 page ;-))
I finished the talk with the ‘catch’ that graylog was – at that time – one of the few Open Core Businesses that got a good split between the Open Source and Closed Source addon. In the last couple of years I did not give that much talks anymore, but still supported customers.

The big difference between other solutions like ELK, was that the Open Source Version of Graylog included a real integration into ADS. So you could decide based on LDAP Groups who could see which kind of messages. The Enterprise version has some very good features too, including Archiving and Auditing.

But with Graylog 4 they removed the possibility of using LDAP Groups, because they restructured LDAP Groups and now declared it an Enterprise feature called Teams.

This is – imho – a very bad business decision. I would like to explain why: one department of my customer has switched to Graylog about 2 years ago. It was free and more and more developers – after some growing pains – really embraced Graylog. After the implementation, it was used more and more and the enterprise version was bought because support was needed and the Archiving and Auditing etc. became more important. Another department in the same company looked also at graylog, but because now they have to buy the enterprise version immediately, they are looking more intensive at other options as well. Now there are also including ELK, Splunk and the integrated EFK Stack in OpenShift in their search. Because all of them are used at some department in the company as well. At the moment it looks that graylog will likely lose at the end and will not be the tool chosen.

To make matters worse – there was not a good communication of change: No warning in the release notes, hey this will go away in the open source version with release XY. No, the releas notes only read: ‘the … old Groups .. have been replaced by Teams in Graylog Enterprise‘. The only good information was in the bug tracker, with other users complaining. I would have hoped for something like that in an official release note: “For Open Source Users of graylog the LDAP and Active Directory group mapping have been removed.

If someone asked me for a Logging Solution in October 2020 my answer would have been Graylog – pure and simple. Now, I will lay out multiple solutions.

Posted in Fedora, logs | Tagged | Leave a comment

Hey, I just need a virtual Machine with …

If you just need a fast (manual) installation with a certain OS. Instead of downloading the ISO and than start the install. Just use virt-install that will do the download, startup etc. for you.

Yes, I know there are better way with images, but if you want an custom installation this is easiest.

virt-install --install centos8 --name  c8
virt-install --install debian10 --name d10
virt-install --install fedora33 --name fed33
virt-install --install centos7.0 --name c7

Yes, it is Centos7.0 for the current Centos7. Centos 7.8 and Centos7 does not work.
With Centos8 and centos-stream8 it uses a better name.

You get a complete list of supported OSs with osinfo-query os.

If you don’t want to use the default disk, memory or CPU you have to specify it.

Here my test for a Secure-boot TPM Install to test Clevis-TPM.

virt-install -n Secure-boot-test --install centos8 --memory=3072 --vcpu=2 --disk size=60,bus=scsi,sparse=true,discard=unmap --disk size=60,bus=scsi,sparse=true,discard=unmap -w network=default --controller scsi,model=virtio-scsi --boot uefi,loader=/usr/share/OVMF/OVMF_CODE.secboot.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/OVMF/OVMF_VARS.secboot.fd,loader_secure=yes --features smm.state=on --tpm model="tpm-tis",backend.type="emulator",backend.version="2.0"
Posted in Enterprise Linux, Fedora | Tagged | Leave a comment