Container Internals

Architecture

Key Numbers

The Seven Namespaces

# Inspect a container's namespaces
$ docker inspect <container> --format '{{.State.Pid}}'
1234
$ ls -l /proc/1234/ns/
lrwxrwxrwx 1 root root 0 May 3 12:34 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 May 3 12:34 ipc    -> 'ipc:[4026532789]'
lrwxrwxrwx 1 root root 0 May 3 12:34 mnt    -> 'mnt:[4026532787]'
lrwxrwxrwx 1 root root 0 May 3 12:34 net    -> 'net:[4026532790]'
lrwxrwxrwx 1 root root 0 May 3 12:34 pid    -> 'pid:[4026532791]'
lrwxrwxrwx 1 root root 0 May 3 12:34 user   -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 May 3 12:34 uts    -> 'uts:[4026532788]'

# Enter a container's namespaces from the host
$ nsenter -t 1234 -a /bin/bash

cgroups v2: One Tree, All Controllers

# Each container gets its own cgroup
$ ls /sys/fs/cgroup/system.slice/docker-<cid>.scope/
cgroup.controllers   memory.current      cpu.weight       io.max
cgroup.events        memory.max          cpu.max          io.stat
cgroup.procs         memory.high         cpu.stat         pids.current
cpu.weight.nice      memory.events       cpuset.cpus      pids.max

# Set CPU and memory at run time
$ docker run --cpus=2 --memory=4g --memory-swap=4g nginx
# Maps to: cpu.max = "200000 100000", memory.max = "4294967296"

# Inspect live usage
$ cat /sys/fs/cgroup/system.slice/docker-<cid>.scope/memory.current
1845231616
$ cat /sys/fs/cgroup/system.slice/docker-<cid>.scope/cpu.stat
usage_usec 12345678
user_usec 8901234
system_usec 3444444

Capabilities and seccomp

# Default Docker capabilities (allowed inside container)
chown, dac_override, fowner, fsetid, kill, setgid, setuid,
setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

# Drop everything, add only what you need
$ docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE my-image

# seccomp profile (default blocks ~50 syscalls)
$ docker run --security-opt seccomp=/path/to/profile.json my-image

# A minimal seccomp.json
{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    { "names": ["mount", "umount2", "reboot", "clone3", "kexec_load",
                  "ptrace", "bpf"],
       "action": "SCMP_ACT_ERRNO" }
  ]
}

# AppArmor / SELinux profile is also applied by default
$ aa-status | grep docker
docker-default ()

OverlayFS: How Layers Work

# An image is a stack of read-only layers.
# A running container adds one writable upper layer.

/var/lib/docker/overlay2/
  <layer1>/diff/    # base OS (e.g., debian)
  <layer2>/diff/    # apt installed packages
  <layer3>/diff/    # COPY app/
  <container>/upper/  # writable layer
  <container>/work/   # overlayfs internal
  <container>/merged/ # what the container sees as /

# Mounted as
$ mount | grep overlay
overlay on /var/lib/docker/overlay2/.../merged type overlay
  (rw,relatime,lowerdir=L1:L2:L3,upperdir=upper,workdir=work)

# Whiteout: deleting a file in upper shadows it in lower
# Implemented as a character device with major/minor 0/0:
$ ls -la upper/
crw-r--r-- 1 root root 0, 0 May 3 12:34 deleted-file

What runc Actually Does

# containerd hands runc a config.json (OCI runtime spec) and a rootfs path
$ runc create --bundle /run/containerd/.../bundle/ my-container
$ runc start my-container

# Roughly, runc does this:
1. clone3() with CLONE_NEWPID|NEWNS|NEWUTS|NEWIPC|NEWNET|NEWUSER|NEWCGROUP
2. Set up the cgroup (write to /sys/fs/cgroup/.../<cid>.scope/cgroup.procs)
3. unshare(CLONE_NEWNS), then mount overlayfs at the new root
4. pivot_root() into the new root
5. mount tmpfs over /proc, /sys, /dev with the right ones masked
6. Apply seccomp filter (prctl PR_SET_SECCOMP)
7. Apply capability bounding set (capset())
8. Apply AppArmor/SELinux label
9. setuid/setgid to the requested user
10. execve(args[0], args)

runc vs crun vs youki

Tradeoffs

Frequently Asked Questions