You might want to look into .nspawn files instead. Then you can also manage your nspawn-containers with the machinectl command.
See man 5 systemd.nspawn
And many command like systemctl and journalctl accept the -M parameter, which allows you to query systemd units inside your nspawn-containers from the host.
edit: The article actually explains all of these things in more detail.
I used to use qemu-user-static to run ARM Linux distros like Buildroot, Yocto, and Raspbian on x88_64. It worked surprisingly well! Outside of some minor bugs here and there, it was perfect for local development, emulating an embedded system I was working on.
If file system level isolation is enough for you, take a loot at schroot (https://linux.die.net/man/1/schroot) which allows root-less chroot. You can use something like debootstrap to get a complete userland into a user controlled directory and use schroot to chroot into it without root level access.
> it seems that for creating a chroot you still require root.
You actually don't as long as you have user namespaces.
One thing I am working on I use chroot (rather unshare --root=) to minimally sandbox a subprocess. At the beginning of the script I have this little snippet:
if [ "$(id --user)" -ne 0 ]; then
exec unshare --map-root-user --mount -- "$0" "$@"
fi
Though you can probably just do something roughtly as `unshare --map-root-user --root=<PATH>`.
Do note that the current support is limited to signed disk images, while it was recently (still not in a release) gained the ability to use any directory that resides inside a signed disk image (instead of just the entire disk image).
There's also https://github.com/termux/proot-distro which may or may not count as containers depending on how you define the word but I think it does count
you can theoretically run a virtual machine like libriscv5 which doesn't require root.
or qemu doesn't require root as well. But qemu is blocked for my usecase. There is flatpak theoretically as well
OCI is for running prepackaged software in black boxes from the internet, where you have no interest or ownership of the container internals.
Most of my containers are not like that. Well, actually none are.
systemd-nspawn is for running your own containers, with a VM-like usage pattern (ie not immutable), deployed as part of your overall systemd based infrastructure for when the thing you need to manage is "too big" to be deployed as its own systemd-service unit, but you still want to be able "to systemd" it.
This distinction is a more useful one that the article made. I love dockerfiles and immutability, but there are good cases for mutable containers, too.
You can also do some neat things with "--ephemeral" and "--volatile" to basically overlay the image (or a subset) with tmpfs; any changes to those overlays will be lost when the container is brought down. The specific mount points can be controlled in greater detail via "--tmpfs" and "--overlay".
Never. If he wanted to go the containers route, Podman is there. There is no reason to use Docker anymore. (Only a satellite tool like docker-compose is not 1-1 compatible with podman-compose, but podman has other ways to orchestrate with systemd as part of podman vision for orchestrating.)
I use nspawn but many of the helpers featured here are new, so I appreciate this article. I've only ever booted from directories rather than images, and wasn't aware that an image could mount its own partitions, even swap!
Also I'm a little unclear on the security implications of "--private-users=id". Yes the user IDs are the same, but it is technically running in a separate user namespace. In terms of security is this mode equivalent to privileged containers, or is it safer?
Redhat's Leapp, for upgrading between major releases of RHEL, uses systemd-nspawn to create a container where it can test installing the packages without interfering with the running OS.
There are lot of ridiculous things in systemd (I'll avoid mentioning specific things to avoid a flame war), but auto containerization of services is by far the most useful thing they've ever come out with. It's a far easier workflow than docker or anything else and is built in "for free"
I've used lots of different container-types over the years to replace VMs with lightweight containers, but right now I'm running systemd-nspawn, and I really, really like it.
The way it integrates with systemd, both inside and outside the container makes it a no-brainer for app-isolation when the app in question is a bit too complex for just being a service-unit in itself, and you don't want to lose observability by hiding everything behind some obscure docker wall.
The way everything integrates into systemctl and you can get aggregated stats for your entire machine and all its sub-containers... Amazingly nice.
I just can't imagine any better way of managing containers on a Linux system than this.
Only thing I would complain about is the name. They really could have come up with something a bit more catchy or self-descriptive. This is probably the only systemd type service which does not immediately shout out what its about, so most people are probably not even aware that systemd can manage containers for you.
Do novel issues get a negative reaction? Retreading old grievances is pointless, but I think if you have a reasonable new gripe (that's not dae hate systemd like me?)you would be just fine.
Most systemd-projects have a name which immediately shouts out what it does, so you can easily tell if it is relevant for your needs or not.
systemd-nspawn is probably the only project without such a name, so most people don't know about it, nor what it does, and therefore never looks any more into it.
And that's a shame really, because it's fantastic technology.
> systemd-nspawn is probably the only project without such a name
Add sd-tmpfiles to the list IMO. While it still create and manages temporary files its more managing almost any type of system file. From creating them to managing their permissions or making symlinks when needed.
I am a strong advocator of renaming it systemd-sysfiles to match the systemd-sysusers which is somewhat related (e.g. tmpfiles using users created from sysusers). But it probably won't happen for a while if at all due to backwards compat.
A debian aarch64 vm on kvm starting a systemd-nspawn for an unpacked raspberry pi 3 iso.
It works way too well judging by how ridiculous it was.
Still saved me a few days instead of setting things up myself.
I actually liked how easy it is to spin up nspawn as a systemd service
See man 5 systemd.nspawn
And many command like systemctl and journalctl accept the -M parameter, which allows you to query systemd units inside your nspawn-containers from the host.
edit: The article actually explains all of these things in more detail.
I am wondering though? Is there something like systemd-nspawn that doesn't require root?
But is there a way to also run OCI compatible directly on this as well?
I don't have root on that system and so I can't create a chroot , there is fakeroot but it doesn't work since it uses qemu on that locked system.
Are there any other alternatives
You actually don't as long as you have user namespaces.
One thing I am working on I use chroot (rather unshare --root=) to minimally sandbox a subprocess. At the beginning of the script I have this little snippet:
Though you can probably just do something roughtly as `unshare --map-root-user --root=<PATH>`.Until then, I'm not sure if there is anything lightweight. If you don't need lightweight, there is Podman.
I am on a completely rootless client at one of my servers.
docker and the rootless nonsense is just root daemons and suid.
...would never have believed marketing lies would reach linux tools if anyone told me this before 2018.
There's also https://github.com/termux/proot-distro which may or may not count as containers depending on how you define the word but I think it does count
yeah you can do some smaller fakechroot and maybe some bind mounts... if you call that a "container" good for you.
There is podman but it requires one time root.
> Hopefully, this article has disabused some of that notion.
If that was the goal, it seems terribly complicated when compared with podman.
Most of my containers are not like that. Well, actually none are.
systemd-nspawn is for running your own containers, with a VM-like usage pattern (ie not immutable), deployed as part of your overall systemd based infrastructure for when the thing you need to manage is "too big" to be deployed as its own systemd-service unit, but you still want to be able "to systemd" it.
This fits my use-case perfectly.
https://0pointer.net/blog/running-an-container-off-the-host-...
I'm not sure how easy that is to customize in Podman.
Also I'm a little unclear on the security implications of "--private-users=id". Yes the user IDs are the same, but it is technically running in a separate user namespace. In terms of security is this mode equivalent to privileged containers, or is it safer?
Surprisingly simple and low footprint solution and genuinely pleasant to work with, since it is very similiar to managing a Systemd service.
[1]https://github.com/JanMa/nomad-driver-nspawn
The way it integrates with systemd, both inside and outside the container makes it a no-brainer for app-isolation when the app in question is a bit too complex for just being a service-unit in itself, and you don't want to lose observability by hiding everything behind some obscure docker wall.
The way everything integrates into systemctl and you can get aggregated stats for your entire machine and all its sub-containers... Amazingly nice.
I just can't imagine any better way of managing containers on a Linux system than this.
Only thing I would complain about is the name. They really could have come up with something a bit more catchy or self-descriptive. This is probably the only systemd type service which does not immediately shout out what its about, so most people are probably not even aware that systemd can manage containers for you.
it's still eating..
https://github.com/mtseet/proot-docker
systemd-nspawn is probably the only project without such a name, so most people don't know about it, nor what it does, and therefore never looks any more into it.
And that's a shame really, because it's fantastic technology.
Add sd-tmpfiles to the list IMO. While it still create and manages temporary files its more managing almost any type of system file. From creating them to managing their permissions or making symlinks when needed.
I am a strong advocator of renaming it systemd-sysfiles to match the systemd-sysusers which is somewhat related (e.g. tmpfiles using users created from sysusers). But it probably won't happen for a while if at all due to backwards compat.
As Brendan Gregg said: "Containers are just processes, cgroups, and namespaces."
You can run unprivileged containers, and in that case, no.