LXC

Installation

Ubuntu

apt install lxc lxc-templates debian-archive-keyring

"debian-archive-keyring" must be installed if containers running Debian should be created.

Directories

• Container: /var/lib/lxc
• Cache for system installations: /var/cache/lxc

Both of these can be links to other directories on another volume. This can make sense, since both directories can easily consume many GiB of disk space. It's important that the destination volume is not mounted with either the "nodev" or "noexec" mount option. If this is the case, it is possible to remount the volume with a command like this:

mount -o remount,rw,nosuid,relatime,uhelper=udisks /dev/AAAb /media/whatever

Additionally, the corresponding line in /etc/fstab shall be adjusted.

Examples

Simple installation of Debian with some useful packages

lxc-create -n deb12c1 -t debian -- -r bookworm --packages nano,screen,iputils-ping,less,dnsutils,bind9-host,wget,nginx,ca-certificates

Troubleshooting key ring errors with debootstrap

Container creation might fail with these messages (Debian Bullseye on Ubuntu 18.04):

debootstrap ist /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-bullseye-amd64 ... 
Downloading debian minimal ...
I: Retrieving InRelease 
I: Checking Release signature
E: Release signed by unknown key (key id 605C66F00D6C9793)
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian
lxc-create: deb11webdev: lxccontainer.c: create_run_template: 1617 Failed to create container from template
lxc-create: deb11webdev: tools/lxc_create.c: main: 327 Failed to create container deb11webdev

The reason is that debootstrap is using the key rings installed in /usr/share/keyrings/. On older Ubuntu systems the key rings for newer Debian releases, like Bullseye, are missing. They can be easily installed using the package from a newer Ubuntu version since there are no dependencies (as can be seen on Ubuntu's package information page debian-archive-keyring (2021.1.1ubuntu1)).

wget http://de.archive.ubuntu.com/ubuntu/pool/universe/d/debian-archive-keyring/debian-archive-keyring_2021.1.1ubuntu1_all.deb
dpkg -i debian-archive-keyring_2021.1.1ubuntu1_all.deb

After that, /usr/share/keyrings/debian-archive-bullseye* exist:

-rw-r--r-- 1 root root 8.5K 2021-03-26 12:11 /usr/share/keyrings/debian-archive-bullseye-automatic.gpg
-rw-r--r-- 1 root root 8.6K 2021-03-26 12:11 /usr/share/keyrings/debian-archive-bullseye-security-automatic.gpg
-rw-r--r-- 1 root root 2.4K 2021-03-26 12:11 /usr/share/keyrings/debian-archive-bullseye-stable.gpg

Configuration

Nested containers

To enable installation of LXC inside a container, it must have the following setting:

lxc.mount.auto = cgroup

Mount file

In the container's config file, add or modify the line specifiing "lxc.mount":

lxc.mount = /var/lib/lxc/container/fstab

The fstab file might look like this:

/host/dir/	/var/lib/lxc/container/rootfs/dir/in/container        none   bind,create=dir

Common problems

Debian Container in Ubuntu until 14.10 or Debian 10 (buster)

Symptoms

Directly after creation of the container it cannot be started. The following error message is being displayed:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied

Or you get a lot of lines in dmesg similar to that one:

[3283539.709422] audit: type=1400 audit(1621956401.478:8529): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-with-nesting" name="/run/systemd/unit-root/" pid=14968 comm="(vnstatd)" srcname="/" flags="rw, rbind"

Or services like Apache2 won't start:

apache2.service: Failed to set up mount namespacing: Permission denied
apache2.service: Failed at step NAMESPACE spawning /usr/sbin/apachectl: Permission denied
apache2.service: Control process exited, code=exited, status=226/NAMESPACE

Solution

Inserting a line in the config-file of the container (usually located in /var/lib/lxc/CONTAINER/config):

lxc.apparmor.profile = unconfined

In LXC version 2 it was

lxc.aa_profile = unconfined

Credits

Container starts without IP address

Probably dnsmasq does not listen on the virtual interface (default: lxcbr0).

Solution: Either getting dnsmasq running or using another virtual interface. In the config-file of the container there is a setting lxc.network.link". virbr0 might be used instead.

lxcbr0 missing at all after installing LXC

Maybe /etc/default/lxc-net is missing.

# This file is auto-generated by lxc.postinst if it does not
# exist.  Customizations will not be overridden.
# Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
# containers.  Set to "false" if you'll use virbr0 or another existing
# bridge, or mavlan to your host's NIC.
USE_LXC_BRIDGE="true"

# If you change the LXC_BRIDGE to something other than lxcbr0, then
# you will also need to update your /etc/lxc/default.conf as well as the
# configuration (/var/lib/lxc/<container>/config) for any containers
# already created using the default config to reflect the new bridge
# name.
# If you have the dnsmasq daemon installed, you'll also have to update
# /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
# Uncomment the next line if you'd like to use a conf-file for the lxcbr0
# dnsmasq.  For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have
# container 'mail1' always get ip address 10.0.3.100.
#LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

# Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc
# domain.  You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR)
# to your system dnsmasq configuration file (normally /etc/dnsmasq.conf,
# or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager).
# Once these changes are made, restart the lxc-net and network-manager services.
# 'container1.lxc' will then resolve on your host.
#LXC_DOMAIN="lxc"

lxcbr0 disappears

There might be a conflict with an existing DNS server. I use bind9 and this helps:

service bind9 stop
service lxc-net restart
service bind9 start

Apache2 not running inside container

For a solution see above (Debian Container in Ubuntu until 14.10 or Debian 10 (buster))!

With my host Ubuntu 18.04 (version of LXC is 3.0.3-0ubuntu1~18.04.1 as of 2019/09/14), Apache won't run inside Debian containers created by these commands:

lxc-create -n deb10c1 -t debian -- -r buster
lxc-create -n deb10c2 -t download -- -d debian -r buster

Obviously this bug is related.

Tips and tricks

Moving containers from one host to another

• Stop the container: lxc-stop -n containername
• In /var/lib/lxc, execute tar --numeric-owner -czf containername.tar.gz containername
• Copy containername.tar.gz from one machine to another (for example, using scp, rsync, wget or curl, or mount a sshfs from the other site)
• On the other machine move the file to /var/lib/lxc
• In /var/lib/lxc, execute tar --numeric-owner -xzf containername.tar.gz
  • Optional: remove the tar file by executing rm containername.tar.gz
• Start container using lxc-start -d -n containername
• Enter container using lxc-attach -n containername -- bash
• Verify that all services are running (for example, using netstat -tulpn or examing the outout of systemctl status (when using systemd) or ps axf)

Troubleshooting

This should not be necessary if "--numeric-owner" has been supplied to the tar command above.

Example: You had mysql installed, but after moving, the service refuses to start.

In the original container, ps axf | grep mysql might print a line like this:

 1531 ?        Sl     0:40  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

Execute that command on the copied container. It might print something like this:

171112 17:50:01 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
171112 17:50:01 [Note] /usr/sbin/mysqld (mysqld 5.5.57-0+deb7u1) starting as process 2140 ...
171112 17:50:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
171112 17:50:01 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
171112 17:50:01 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
171112 17:50:01 InnoDB: The InnoDB memory heap is disabled
171112 17:50:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
171112 17:50:01 InnoDB: Compressed tables use zlib 1.2.7
171112 17:50:01 InnoDB: Using Linux native AIO
171112 17:50:01 InnoDB: Initializing buffer pool, size = 128.0M
171112 17:50:01 InnoDB: Completed initialization of buffer pool
171112 17:50:01  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

If you execute ls -lh ./var/lib/mysql/mysql/plugin.frm, an output similar to that one might appear:

-rw-rw---- 1 106 110 8,4K Aug 14 09:28 ./var/lib/mysql/mysql/plugin.frm

• What's the problem? At the position that should show the owning user and owning group you see "106" and "110" instead.
• How could that happen? The tar command was unable to correctly resolve the owner information. This is a common pitfall in Linux and other Unix-like systems: the user "mysql" can have another uid on every system (as id -u mysql would show). Furthermore, the system (the LXC host) that ran the tar command, did not have a user "mysql".

By "cd /", go to the file system's root directory and execute these commands:

find -uid 106 -exec chown mysql {} \;
find -gid 110 -exec chgrp mysql {} \;

If the above "ls" command printed other uid/gid values, replace as needed.

Container migration after upgrade from LXC 2.x to 3.x

lxc-update-config can be used (Credits).

Links

• Repository "lxcscripts" from Michael Kremser at GitLab: contains various useful scripts
• Container configuration man page
• LXC in ubuntuusers.de (German)
• LXC in Debian Wiki
• LXC Template for Devuan
• Containers: Just Because Everyone Else is Doing Them Wrong, Doesn't Mean You Have To