Then they removed that in favor of Quadlet. Now in order to do a single container I can do a unit file, but for a pod, I need to use a Kubernetes cluster definition.

Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.

So what is it, Podman? Should I just use Kubernetes? Should I just make dedicated directories for everything instead of mapping logical places for things?

To enable the feature:

    $ podman-compose systemd -a create-unit

    $ podman-compose systemd -a register
    $ systemctl --user enable --now "podman-compose@$PROJECT_NAME"

    $ podman-compose pull
    $ systemctl --user restart "podman-compose@$PROJECT_NAME"

Source code for the feature is here if you care to inspect it:

https://github.com/containers/podman-compose/blob/f6dbce3618...

Admittedly, I'm still on podman 4.3.1, but I don't see any reason why this would stop working in later versions of podman.

    usage: podman-compose [-h] [-v] [--in-pod in_pod] [--pod-args pod_args]
                          [--env-file env_file] [-f file] [-p PROJECT_NAME]
                          [--podman-path PODMAN_PATH] [--podman-args args]
                          [--podman-pull-args args] [--podman-push-args args]
                          [--podman-build-args args] [--podman-inspect-args args]
                          [--podman-run-args args] [--podman-start-args args]
                          [--podman-stop-args args] [--podman-rm-args args]
                          [--podman-volume-args args] [--no-ansi] [--no-cleanup]
                          [--dry-run]
                          {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
                          ...
    
    command:
      {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
     ...
        systemd             create systemd unit file and register its compose stacks
                            
                                When first installed type `sudo podman-compose systemd -a create-unit`
                                later you can add a compose stack by running `podman-compose systemd -a register`
                                then you can start/stop your stack with `systemctl --user start podman-compose@`

The --help is fine documentation for the people who have already installed the tool, but it doesn't help people like OP who just want a simple way to run multiple containers as a systemd unit and don't yet know that podman-compose has a solution.

That's why I said "poorly documented" instead of "undocumented". It's there once you know where to look.

the concern with truly undocumented functionality is that it's not included intentionally: either it's a bug or an experimental feature that could be removed or changed with no notice. a poorly-documented feature, on the other hand, will probably at least get a deprecation notice before it disappears

Which is why I didn't call it that, I called it poorly documented.

But even for small-scale single-node production use cases, I suspect that podman-compose with systemd doesn't have the same concerns as docker-compose does. Since you're registering the workload with systemd, it'll restart with the node as easily as any other service, and rootless containers are a big win for security.

Where you can't keep using (podman|docker)-compose is when you have to scale up a service beyond a single node.

That's a bug in docker. If your system isn't configured for SELinux, disable it.

Also the systemd files generated by podman-generate-systemd are just executing "podman start containername", you can write them on your own easily but (unlike e.g. docker-composr) the containers are black boxes pretty much.

The advantage of quadlet is that the definition of the container is declared in the .container file; before I used to write the podman run command line manually in a handwritten systemd unit, and quadlet is a big improvement in that respect and can be an alternative to docker-compose (with advantages and disadvantages).

I feel obligated to say that you should set it to permissive mode, never disable SElinux.

    sudo setenforce 0

It's likely to little gain. I know why people don't, but it is very accessible to those open to it

The more one adheres to the FHS, the easier SELinux is

Terrible documentation, terrible mental model, terrible CLI UX, terrible error messages.

I run Fedora and SELinux is working well enough, but it's a piece of machinery I can't wait to see replaced, however useful people swear it is.

For systems that shouldn't do much more than exactly what is prescribed, it's acceptable, is what I'm after... I guess.

I can't do justice to the source, but there's a concept about our programs/creations reflecting us.

Like a peer hints - SELinux reflects an agency like the NSA, draconian. Good and bad

I’m not really sure what this means (I have 0 knowledge on selinux)

SELinux has a bit of a well deserved reputation... but I, a fairly silly person, have managed to work with it

This video likely explains things far better than I can in this post:

https://www.youtube.com/watch?v=_WOKRaM-HI4

I'll probably fail with specifics, where they certainly do a better job.

So. First it's important to know SELinux runs in one of two modes:

    * A targeted mode where well-known/accounted-for things are protected. For example, nginx
    * A more draconian mode where *everything* is protected

The protection is based on policies that say 'things with this label/at this path are allowed to do XYZ'.

It's very focused on filesystem paths and what relevant applications try to do.

It's entirely manageable, but admittedly, complicated. Without practicing the words I can't express them.

Most people having trouble with SELinux are defying some convention. For example: placing application scratch data in '/etc'.

Policy management is a complicated topic.

The policy can be amended in cases where the standard doesn't apply; I won't cast judgement - sometimes it's a good idea, sometimes not.

Another way to handle this is to copy the label from one path and apply it to the one your application requires/customizes. This is less durable than leaning on the policy.

It acts as a sort of central DB... the goal is to make things such that the policy stores all of the contexts so the files/dirs can have "labels" applied for SELinux

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

They provide guidance on how a given filesystem path should be used.

This has informed the default SELinux policies greatly; familiarity turns hassle into informed assumptions/ease.

SELINUX=permissive

I keep forgetting this transition happened until I try to 'podman generate systemd [...]'

This is rare because I wrote an Ansible role to do this in a way that feels nice.

Anyway, it really feels like podman lost the mark. I've already subscribed to the unit file maintenance/relationship planning thing. Just let me use the generator. I don't care about Quadlets or how they might be better.

Add the following to containers.conf if you don't want to deal with it:

   [containers]
   label=false

    podman run -v .:/app:z image

https://github.com/aksiksi/compose2nix

Arion can wrap docker-compose and run as a project or part of a nixos config. Did you come across Arion before creating this, and have you compared them?

I had a brief look through your examples and it doesn’t look like compose2nix implements docker-compose’s network per compose file. Is this something you want to add?

My tool does the opposite: it takes a Compose file and converts it into OCI containers in Nix. The idea is that your Compose file is the source of truth, and you simply generate Nix to run on NixOS. One benefit here is that you can easily migrate an existing Compose project into native Docker/Podman containers running on NixOS. This removes Docker Compose from the equation entirely - essentially a “reimplementation” of Compose.

It should support the default network per Compose project. See: https://github.com/aksiksi/compose2nix/blob/6dc451fd960f7a9b.... If you’re talking about something else, please feel free to open an issue and I can take a look.

    [Unit]
    Description=Jellfyin
    After=NetworkManager-wait-online.service
    
    [Service]
    Restart=always
    ExecStart=/usr/bin/podman run --rm --volume '/srv/jellyfin/config:/config:Z' --volume '/srv/jellyfin/cache:/cache:Z' --volume '/media/media:/media:Z' --publish '8096:8096' --publish '8920:8920' --publish '1901:1900/udp' --publish '7359:7359/udp' --name 'jellyfin' 'docker.io/jellyfin/jellyfin:latest'    
    ExecStop=/usr/bin/podman stop -t 5 jellyfin ; /usr/bin/podman rm -f jellyfin
    
    [Install]
    WantedBy=local.target

Isn't that great? You want SELinux if you deploy containers on prod.

> I have repeatedly struggled with containers unable to access mapped directories.

Use the `z` or `Z` flag, e.g. `podman run -v mydir:podman_dir:z nginx` to get SELinux labels updated automatically (and temporarily) by podman.

Mapping directories from the host requires that you change selinux labels on those files so that the container process can access the files. That's just how selinux works.

It's just a simplified layer ontop of those old systemd units.

feels like a lot more cruft than quadlets to me though.

I've also had a lot of VPNs break and/or be broken by Docker. I don't know much about the way podman does networking, but whatever it is they did a good job thinking it through and it has yet to interfere with anything else I do. I definitely can't say the same for Docker

To see why, look at Buildah without `bud`. Instead of a silly DSL that becomes annoying the second your use-case veers an inch off the happy path, you can use Bash, or Fish, or whatever it is that you want.

These types of bad decisions carry the into the rest of the Docker ecosystem. DCS and it's (perpetually incomplete) replacement are yet more examples: instead of using established signing protocols (like Cosign does) they had the desire to build an obnoxiously complex and hard to automate (especially key rotation) system.

I'm doing it right now, no nightmares, just works, odd.

Very cool.

With Podman (and Docker rootless), subuids/subgids[1] are _usually_ used. Root in the container is root only within the Linux user namespace. You can map users to real users, including yourself and root - which is how distrobox works, but it's secure by default: your subordinate users can't even access your resources.

The primary limitation of rootless is networking, the kernel doesn't have a built-in way to do rootless networking in the way that you want for containers.

[1]: https://www.man7.org/linux/man-pages/man5/subuid.5.html

To be perfectly honest I do enjoy it for all my personal container use. But at work I still use docker for our developers. There is so far nothing I can offer our developers that can match docker compose in simplicity.

We even use buildah in CI pipelines when we make container images, but specifically for developer end users docker compose is still dominant.

I almost got burned by this.

The benefit with sticking with docker is that accessibility is better: More communities, more blogs, broad availability of docker compose configs, more peers knowing how to use it etc. In the end, both podman and docker run processes in isolated namespaces on the host.

[1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...

The main issue to a common developer would be Docker compose I suppose, which if you use simple compose files, there's actually a podman-compose script that attempts to be compatible with the Docker compose spec.

There's also using Podman as a backend for docker-compose [1]. Overall, in 2024, I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.

[1] https://www.redhat.com/sysadmin/podman-docker-compose

They originally tried to work together with docker to fix various issues (e.g. systemd compatibility for certain use case) which had popped up in the past but this turned out not very fruit full.

If you combine this that docker had and still has a uncanny amount of security issues(1) and docker like containers/images being very widely used by developers they had very little choice then to create their own implementation which is more in line with the values and approaches of RHEL.

(1): Like docker still not defaulting to rootless even through it can. For a hardened context both the the docker user group and the ways to run it without the group are security wise a no-go in many use-cases. Like docker initially doing way to little to make sure non privileged containers are at least somewhat sandboxed and taking forever to fix it even when it became known it's an issue. Like the way it interacts with firewalls and networks rules. Like the way it interacts with SELinux. Etc. In companies with dedicated Linux system administrators which care about security docker being banned is not that rare.

Enter Podman, you just winget install podman and it gets out of your way. When you need to run a container, you start up the Podman VM. When you don't, your system behaves exactly as it always did before. If you need to run something in a compose file, podman-compose is there. Maybe it can't handle esoteric configs, but it's worked for my use cases. The only thing Podman doesn't do is integrate with VS Code properly, but honestly the struggle of getting Docker to behave itself on Windows is far more annoying to me than missing some keyboard shortcuts in VS Code, so for me it's not a problem.

I would (and do) recommend Podman as the default solution for containers on Windows. The only reason I can see to use Docker is if your company is paying for it, or if you have a complicated compose setup, in which case it might be worth moving to Kubernetes anyway.

I hope the majority doesn't end up with your point of view. Docker is not RedHat/IBM. If Docker goes away RedHat gets to continue to push their corporate agenda with a heavier hand. There are some advantages to Podman, but there are also some things that have not been executed well in RedHat's mission to replace all things Docker. If you look at the history of how RedHat approached the container space they could have contributed, and improved, a lot of projects they ultimately wasted time and effort on. RedHat isn't a company that has an agenda of doing right by the customer and that's why you should consider what you hope for from a bigger picture perspective.

Red Hat tried to work with Docker, but it didn’t go well. Docker also shot itself in the foot trying to push Swarm over Kubernetes and a bunch of other silliness until they had some serious management/leadership changes.

Podman, IIRC, is 100% FOSS. Not sure you can say the same for Docker. If Red Hat gets stupid with Podman, the rest the community can pick it up and carry on under a new name. Not true of Docker. If Docker gets bought by, say Broadcom tomorrow, the community can only fork the bits that are open.

they tried to work with docker, it didn't work out, that is why we have podman

and given the headache "securely" using docker is I'm really happy that they did

After the first two or so sentences, you don't mention a single specific thing in your comment, so it's difficult to understand what you mean.

they just never bothered making it the default or officially supporting it

which in context with how they acted about other security problems int the past tells a lot about how serious the docker company takes security on linux

Podman Desktop is completely open.

https://github.com/docker/docs/issues/6910

I did not know this as I stopped using Docker long ago.

And my point was more so that podman is obviously designed around the rhel ecosystem. I'm not saying it's closed! Just that even if we were to (wrongly) argue that one of the two is more "locked in", it's clearly podman. Docker is so much more widely used, ported, is basically as completely "non locked in" as it could be.

The only possible "lock in" is maybe the docker images namespace defaulting to docker hub but imo that's trivial and basically more of an early design choice that can't be reverted.

By all means, we can argue about technical differences but the often repeated argument about docker being less open than podman or whatever is just not true

And this isn't even a Podman-specific issue, it's true of Docker Desktop as well [0]:

> The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

[0] https://docs.docker.com/network/drivers/host/

I started up Podman Desktop, and did a terminal command "docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main" based on Github's instructions, but it gave a error message something about "host".

the exact error message was ""Error": "failed to create new hosts file: unable to replace \"host-gateway\" of host entry \"host.docker.internal:host-gateway\": host containers internal IP address is empty""

Do you know what is the problem and how do I overcome this?

If I run the above command using Docker Desktop, it runs and installs Ollama-WebUI just fine.

Thank you.

(1) https://github.com/ollama-webui/ollama-webui ("Installing with Docker")

It would be good to have a swarm alternative in podman, it seems like k8s is a crutch for lack of orchestration. With their good security hat on they could probably come up with a sane, simple way of running containers at small scales without having to dive into a PhD in k8s which doesn't have secure defaults out of the box, while maintaining compatibility with docker compose format.

Anyway that's a good intro thanks for sharing, I'll be trying it later.

More than once I wrecked my whole setup running system migrate as a way to fix problems. It has a whole thing about security ACLs, ID mapping and labels. A chmod -R under your home folder will probably kill all your containers.

While I'm happy with the results it was far from an "it just works" solution like Docker. I imagine things probably have been improved since I started using it.

I’ll give it a shot again. Looks like it has matured very fast.

https://gist.github.com/acdha/9be1c3521af4f18d9f86264a889581...

Everything is nicely explicit and allows for a good mental model of what's going to happen when you run it.

    source "/path/bwrap_helper.sh"
    FLAGS=(
      ${FLAGS_ROOTFS_DISTROX_MIN[@]}
      ${FLAGS_ENV_XDG_GUI[@]}
      ${FLAGS_PULSE[@]}
      ${FLAGS_GPU_ACCEL[@]}
      --new-session
      --bind /path/jail123 /home/user
    )
    exec bwrap "${FLAGS[@]}" --seccomp 10 10

*(I say "used to be" because Linux is now even more essential and central, not less.)

I'm going to assume that "bigger privately-owned brother(s)" you're referring to Docker? If so, ironic given all of the evil things RedHat and IBM do in the OSS realm.

I just keep hoping that Docker isn't that bad and is a good alternative to Podman, because I've read mostly good things about the it, while Docker usually gets dragged through the dirt.

Docker tooling tends to work with podman but you have to do a bunch of stuff like run a server to imitate docker. At that point why bother?

And no, reverse proxies do not solve this problem; lots of protocols (e.g. SSH) have no equivalent to X-Forwarded-For for identifying the remote host.

So updating a purportedly single rule might actually require to update several underlying traffic mangling rules, with logic that is not readily apparent. Or even easy to reason about. When you add the ability to route traffic directly from container to container without passing through the outermost interface, things can get quite hairy.

That daemon is a subtle but incomensurate burden when adopting the least privilege principle.

In environments running multiple hosts which themselves run multiple containers, typically: k8s, it forces your hand in either giving in and grant docker (the daemon) root privilege if any one of your container needs root, or to exclude that container from running in that environment altogether (since it would fail to execute if the docker daemon is in rootless mode). Of course the most secure and wise option would be to refractor that container and whatever it's doing to run rootlesssly, but sometimes this is simply not a reasonable or even possible option in order migrate massive complex platform of hundreds of micro services with its own history and justified security exceptions.

K8s (and Openshift, which adopts a stronger security by default configuration set) provides control over which service accounts is granted such exception.

tl-dr: use podman/buildah rather than docker, use openshift rather than vanilla k8s.

- They by default don't.

- It's only semi officially supported. It requires non official tooling, through they link to that tooling officially in their doc. But issues specific to rootless docker seem to not be much of a priority.

- roots less docker sometimes has some slight issues, but mostly minor stuff

The fact that they can make it save to use, but do not, is a really huge red flag. And it's not the first time they didn't take security on linux not serious at all.

To clarify what I mean with "make it save", the docker user group allows easy gaining root light access, which is a huge security no-go. And the alternative is using sudo or similar all the time which also is a security no-go (if you e.g. have a dev system, it's okay for starting docker images as services, but then limitations with systemd integration make podman often a better choice here anyway). And while there are ways to make it work without a security no-go and the daemon the last time I checked they weren't out-of the box and in my opinion too brittle.

Now if you have a single user system you maintain yourself and have a single user+admin+sudo right user or similar maybe then you don't care about docker group or using sudo too often. But if it's a company managed system with reasonable security requirements it's an absolute no go.

I see they've added some k8s integration, but not sure how you see it being a k8s alternative? Maybe a minikube alternative...

it misses some features docker has and docker misses some features podman has and some subtle behaviour differences can lead unexpected outcomes (mostly in favor of podman doing things better)

many of the differences are irrelevant but can be an issue in scripts calling docker/podman cli

one of the more surprising discrepancies we ran into was that due to the rootless podman nature you don't need to "slowly copy the context to the daemon". In our case this lead to some slightly sub-optimal setups running just fine with podman but then taking way to long to run with docker as it copied a ton of data.

generally in my experience if you only use podman or only use docker and the "drop-in" aspect only matters for switching from one to the other permanently it works grate, but if you idk. want to use docker on some systems and podman on others it might not be the best idea