将overlay2存储驱动程序与覆盖文件系统结合使用

Question

目标

我在一张活动CD中运行docker，我想缓存docker构建的结果，主要是当我重新启动到这个活动CD中时。我的想法是在/var/lib/docker中设置一个overlayfs。因此，我在/etc/fstab中有以下内容：

overlay /var/lib/docker overlay lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work,x-systemd.requires=/mnt/root,x-systemd.requires=/var/lib/docker,nofail 0 2

但是，我也希望使用overlay2存储驱动程序。我很难让双方合作。我尝试了两种方法来实现这一点。

方法1(mount之前的docker服务(

1a。启动docker.{socket,service}系统单元。

sudo systemctl restart docker.service # restarts docker.socket automatically

1b。启动overlayfs

sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work

这两个步骤都执行成功。但是，如果我尝试构建/运行一个新映像，则会出现以下错误。

故障类型1

$ make up
mkdir -p .dynamodb-data/
docker-compose up -d --build
Pulling dynamodb (amazon/dynamodb-local:)...
latest: Pulling from amazon/dynamodb-local
2cbe74538cb5: Pull complete
2b0e6045b331: Extracting [==================================================>]    150MB/150MB
667cfda26bc2: Download complete
ERROR: failed to register layer: Error processing tar file(exit status 1): failed to mknod("/etc/sysconfig/clock", S_IFCHR, 0): operation not permitted
make: *** [Makefile:15: up] Error 1

故障类型2

$ docker run -it --rm alpine                         
docker: Error response from daemon: open /var/lib/docker/overlay2/011d016883746bd72b20f298ecd73545a982019acd9eb3f600ff3b2bd4768888/committed: no such file or directory.
See 'docker run --help'.

方法2(docker服务之前的mount(

2a。启动overlayfs

sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work

2b。启动docker.{socket,service}系统单元。

sudo systemctl restart docker.service # restarts docker.socket automatically

2a。成功(当然(，但2b。失败。我尝试捕获下面的一些systemd日志，试图帮助诊断这个问题。

故障类型1

正在运行命令

$ sudo systemctl restart docker              
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.

journalctl

$ journalctl -u docker
-- Journal begins at Sat 2022-01-08 12:36:55 EET, ends at Sat 2022-01-08 13:30:15 EET. --
Jan 08 12:37:11 johnos systemd[1]: Starting Docker Application Container Engine...
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.548295176+02:00" level=info msg="Starting up"
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575509563+02:00" level=info msg="libcontainerd: started new containerd process" pid=2346
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575558875+02:00" level=info msg="parsed scheme: "unix"" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575566468+02:00" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575590744+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575605990+02:00" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.212959027+02:00" level=info msg="starting containerd" revision=v1.4.11 version=v1.4.11
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229766943+02:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229835283+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233082343+02:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.aufs"..." error="aufs is not supported (modprobe aufs failed: exit status 1 "modprobe: FATAL: Module aufs not found in directory /r>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233119579+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233257178+02:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.btrfs"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (tmpfs) must be a btrfs filesystem to be>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233270945+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.devmapper"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233289127+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233297875+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233320748+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233391289+02:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233485569+02:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233500101+02:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233516933+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233524181+02:00" level=info msg="metadata content store policy set" policy=shared
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233604310+02:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233619120+02:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233663958+02:00" level=info msg="loading plugin "io.containerd.service.v1.introspection-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233690208+02:00" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233702617+02:00" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233713672+02:00" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233728996+02:00" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233740040+02:00" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233751585+02:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233763065+02:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233773387+02:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233828995+02:00" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233872779+02:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234064302+02:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234082499+02:00" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234118718+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234131212+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234141269+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234150671+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234160162+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234172118+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234181945+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234191381+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234200863+02:00" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234238659+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234251883+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234262989+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234272388+02:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234387869+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234413901+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234435704+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234447184+02:00" level=info msg="containerd successfully booted in 0.021879s"
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237690927+02:00" level=info msg="parsed scheme: "unix"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237706112+02:00" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237717947+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237724235+02:00" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238078323+02:00" level=info msg="parsed scheme: "unix"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238095004+02:00" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238105606+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238112459+02:00" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.293257377+02:00" level=info msg="Loading containers: start."
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.360740919+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 08 12:37:14 johnos dockerd[2267]: time="2022-01-08T12:37:14.902691695+02:00" level=info msg="Loading containers: done."
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113537187+02:00" level=info msg="Docker daemon" commit=v20.10.12 graphdriver(s)=overlay2 version=20.10.12
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113579097+02:00" level=info msg="Daemon has completed initialization"
Jan 08 12:37:15 johnos systemd[1]: Started Docker Application Container Engine.
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.121263835+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.124035385+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:42:57 johnos systemd[1]: Stopping Docker Application Container Engine...
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199583930+02:00" level=info msg="Processing signal 'terminated'"
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199951358+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.200178572+02:00" level=info msg="Daemon shutdown complete"

Answer 1

内核不支持覆盖之上的覆盖。没有docker配置可以改变这一点。您可以在Docker的文档中找到支持的备份文件系统的列表。如果您的备份文件系统是overlay，那么最不坏的选择是fuse overlayfs，它在用户空间中运行。最简单但也是最糟糕的选择是vfs，也就是本机存储驱动程序，它是每个层和容器的文件系统的完整副本(根本没有覆盖(。