Podman Slow Launch With Userns Id Mapping

When using a --userns argument, such as --userns=keep-id:uid=1000,gid=1000 Podman can hang when launching your container. This is caused by a remapping that must occur in the layers of the image, the larger the image the longer it takes.

Auditd

If you are on a STIG’d box I have found that the following rules dramatically impact the runtime of various podman commands including prunes.

-a always,exit -F arch=b32 -S chown,fchown,fchownat,lchown -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S chown,fchown,fchownat,lchown -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S rename,unlink,rmdir,renameat,unlinkat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b64 -S rename,unlink,rmdir,renameat,unlinkat -F auid>=1000 -F auid!=unset -k delete

They can be replaced with these lines to exclude podman

# Exclude container runtime processes from chown operations
-a never,exit -F arch=b32 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/podman
-a never,exit -F arch=b64 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/podman
-a never,exit -F arch=b32 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/conmon
-a never,exit -F arch=b64 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/conmon
-a never,exit -F arch=b32 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/crun
-a never,exit -F arch=b64 -S chown,fchown,fchownat,lchown -F exe=/usr/bin/crun

# Log all other chown operations
-a always,exit -F arch=b32 -S chown,fchown,fchownat,lchown -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S chown,fchown,fchownat,lchown -F auid>=1000 -F auid!=unset -k perm_mod

# Exclude container runtime processes from delete operations
-a never,exit -F arch=b32 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/podman
-a never,exit -F arch=b64 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/podman
-a never,exit -F arch=b32 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/conmon
-a never,exit -F arch=b64 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/conmon
-a never,exit -F arch=b32 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/crun
-a never,exit -F arch=b64 -S rename,unlink,rmdir,renameat,unlinkat -F exe=/usr/bin/crun

# Log all other delete operations
-a always,exit -F arch=b32 -S rename,unlink,rmdir,renameat,unlinkat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b64 -S rename,unlink,rmdir,renameat,unlinkat -F auid>=1000 -F auid!=unset -k delete

OverlayFS

Most people are familiar with OverlayFS, the native Overlay in the kernel is only available to root though, so when running rootless we get thousands of chown commands which cause our slowdown. For rootless we can configure OverlayFS to be used instead at the user level, see issue 17933 for why that cannot be easily configured at the system level.

~/.config/containers/storage.conf

[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

To globally configure this at a user level you can add the following into a profile.d script export STORAGE_OPTS="overlay.mount_program=/usr/bin/fuse-overlayfs"

Issues to follow

• WIP podman: propagate user namespace mappings to image pull by giuseppe · Pull Request #28393 · podman-container-tools/podman
• —userns=keep-id storage-chown-by-maps kills machine with large images · Issue #16541 · podman-container-tools/podman

---

Sources:

• https://github.com/podman-container-tools/podman/issues/11220
• https://github.com/podman-container-tools/podman/issues/26479
• https://discussion.fedoraproject.org/t/extremely-slow-rootless-quadlet-start/155650
• https://github.com/podman-container-tools/podman/issues/17933
• https://github.com/podman-container-tools/buildah/issues/3120