Gold and silver cogwheel collection

Install latest Docker binaries on Synology NAS

In this post we are going to install the latest Docker binaries on a Synology NAS without using the official Synology Docker package. Because the Docker version that is currently installed by the this package is pretty old, upgrading to a newer version is a hot topic in the Synology community.

There are scripts available on the internet for upgrading the Docker version that is installed by the Synology Docker package, but they all fail when using the Btrfs file system. To get to the heart of the problem, I will install the Docker binaries without using the Synology Docker package on a clean installation of DSM.

For this test I am using a vanilla installation of DSM 7.0-41222 Beta on VirtualDSM and the latest Docker binaries from the Docker website. At the time of writing the latest Docker version is 20.10.5 (2021-03-02) and the Docker version installed by the Synology Docker package is 18.09.8 (2019-07-17).

We are going to follow the instructions in the Install Docker Engine from binaries page and because the Synology NAS is based on Linux, we move directly to the section Install daemon and client binaries on Linux.

Please note that the commands executed below can irreparably damage your Synology NAS and cause data to be permanently lost. So be very careful when trying it out yourself.

Prerequisites

Since Synology is also releasing a working Docker package, as expected, we meet the prerequisites set in the instructions.

root@DS:~# uname -a
Linux DS 4.4.180+ #41222 SMP Fri Dec 4 19:02:11 CST 2020 x86_64 GNU/Linux synology_kvmx64_virtualdsm
root@DS:~# iptables -V
iptables v1.8.3 (legacy)
root@DS:~# ps -V
ps from procps-ng 3.3.15
root@DS:~# xz -V
xz (XZ Utils) 5.2.1
liblzma 5.2.1
root@DS:~# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  7       1       1
cpu     4       1       1
cpuacct 6       1       1
blkio   8       1       1
memory  5       1       1
devices 3       1       1
freezer 9       1       1
root@DS:~#Code language: Markdown (markdown)

Install Docker daemon and client binaries

Download the static binary archive. Go to https://download.docker.com/linux/static/stable/ (or change stable to nightly or test), choose your hardware platform, and download the .tgz file relating to the version of Docker Engine you want to install.

In this case we are using the latest stable version 20.10.5 for hardware platform x86_64.

root@DS:~# wget -q https://download.docker.com/linux/static/stable/x86_64/docker-20.10.5.tgz
root@DS:~# ls
docker-20.10.5.tgz
root@DS:~#Code language: Markdown (markdown)

Extract the archive using the tar utility. The dockerd and docker binaries are extracted in a directory docker.

root@DS:~# tar xvzf docker-20.10.5.tgz
docker/
docker/docker-init
docker/docker
docker/containerd-shim
docker/ctr
docker/docker-proxy
docker/runc
docker/containerd-shim-runc-v2
docker/containerd
docker/dockerd
root@DS:~#Code language: Markdown (markdown)

For now we are not going to move the binaries to /usr/local/bin as described in the instructions, but we are going to adjust the PATH variable so the binaries can be found when we want to execute them.

root@DS~# PATH=/root/docker:$PATH
root@DS:~# echo $PATH
/root/docker:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin
root@DS:~#Code language: Markdown (markdown)

Because by default we use the aufs storage driver, I guess we also need the auplink binary. For now I will copy this binary from an existing Synology NAS with the Synology Docker package installed.

root@DS:~# scp sysop@DS716:/usr/local/bin/auplink docker/.
admin@192.168.0.111's password:
auplink                                       100%   14KB   9.8MB/s   00:00
root@DS:~#Code language: Markdown (markdown)

Start Docker daemon

At this moment, we only care about errors. That is why we increase the logging level to “error” before starting the Docker daemon.

root@DS:~# dockerd --log-level error
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.3 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)
root@DS:~#
Code language: Markdown (markdown)

The error message indicates that we perhaps need to upgrade iptables or the kernel. If we take a closer look at the messages, we probably are missing some kernel modules.

Insert kernel modules

The code below is from the start-stop-status script that is provided by the official Synology Docker package. Let’s see if this will solve the problem and execute the code. This code will insert the necessary modules into the kernel.

source /usr/syno/etc.defaults/iptables_modules_list
DockerModules="xt_addrtype.ko xt_conntrack.ko veth.ko macvlan.ko aufs.ko"
DockerBridgeModules="llc.ko stp.ko bridge.ko macvlan.ko"
if [ -f /lib/modules/br_netfilter.ko ]; then
    DockerBridgeModules="${DockerBridgeModules} br_netfilter.ko"
fi
DockerIngressModules="iptable_mangle.ko xt_mark.ko ip_vs.ko ip_vs_rr.ko xt_ipvs.ko"
DockerServName="docker"
InsertModules="${KERNEL_MODULES_CORE} ${KERNEL_MODULES_COMMON} ${KERNEL_MODULES_NAT} ${IPV6_MODULES} ${DockerModules} ${DockerBridgeModules}"
if [ -f /lib/modules/ip_vs.ko -a -f /lib/modules/ip_vs_rr.ko -a -f /lib/modules/xt_ipvs.ko ]; then
    InsertModules="${InsertModules} ${DockerIngressModules}"
fi
iptablestool --insmod "${DockerServName}" ${InsertModules}
Code language: Bash (bash)

let’s try to start the docker daemon again. Using the ampersand (&) will start the process in the background.

root@DS:~# dockerd --log-level error &
[1] 13924
root@DS:~#Code language: Markdown (markdown)

Run hello-world

Now the Docker daemon does start without errors. To test if everything works correctly, we will run the hello-world container that displays a welcome message.

root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu markdown

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@DS:~#
Code language: Markdown (markdown)

At this point we have a working Docker installation. Because the Synology Docker GUI is not available, as an alternative, Portainer could be used as a convenient container management tool.

Although the Docker installation works, the Docker files are stored on the root file system where the available space is limited. Also, the storage driver aufs is deprecated as we will see in the next section.

Storage drivers

Storage driver aufs

AUFS is a union filesystem. The aufs storage driver was previously the default storage driver used for managing images and layers on Docker for Ubuntu, and for Debian versions prior to Stretch. If your Linux kernel is version 4.0 or higher, and you use Docker Engine – Community, consider using the newer overlay2, which has potential performance advantages over the aufs storage driver.

Let’s have a closer look at the docker info output.

root@DS:~# docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.5
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: extfs
  Dirs: 0
  Dirperm1 Supported: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
 Kernel Version: 4.4.180+
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.859GiB
 Name: DS
 ID: PO3D:IIPE:M6KW:TFPS:6JPY:DP4T:ZUOY:AVGP:KCEC:6CFG:OQLM:KFE3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.
root@DS:~#Code language: Markdown (markdown)

As you can see the storage driver is aufs, and there is also a warning at the bottom that the aufs storage-driver is deprecated, and will be removed in a future release.

Storage driver overlay2

OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. Docker provides two storage drivers for OverlayFS: the original overlay, and the newer and more stable overlay2.

First, let’s stop the Docker daemon and try another storage driver.

root@DS:~# ps -ef | grep dockerd
root     13924 12415  8 16:14 pts/1    00:00:00 dockerd --log-level error
root     14167 12415  0 16:14 pts/1    00:00:00 grep --color=auto dockerd
root@DS:~# kill 13924
root@DS:~#Code language: Markdown (markdown)

Docker supports a number of storage drivers. Because overlay2 was suggested earlier as an alternative to aufs, let’s give that a try.

root@DS:~# dockerd --storage-driver overlay2 --log-level error
ERRO[2021-03-30T16:20:45.969233496+02:00] failed to mount overlay: no such device       storage-driver=overlay2
failed to start daemon: error initializing graphdriver: driver not supported
root@DS:~#
Code language: Markdown (markdown)

Using this driver does not appear to work. Furthermore, the Docker files are stored on the root filesystem which is limited in space. It is better to store the Docker files on /volume1. Because the file system on /volume1 is Btrfs, let’s focus on the storage driver btrfs.

Storage driver btrfs

Btrfs is a next generation copy-on-write filesystem that supports many advanced storage technologies that make it a good fit for Docker. Btrfs is included in the mainline Linux kernel.

Docker’s btrfs storage driver leverages many Btrfs features for image and container management. Among these features are block-level operations, thin provisioning, copy-on-write snapshots, and ease of administration. You can easily combine multiple physical block devices into a single Btrfs filesystem.

Start the Docker deamon with storage driver btrfs.

root@DS:~# dockerd --storage-driver btrfs --log-level error
failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)
root@DS:~#
Code language: Markdown (markdown)

This actually makes sense because the root filesystem is ext4 and not Btrfs. In other words: we are using the btrfs storage driver on the wrong filesystem.

Let’s point the data root to /volume1/@docker which is located on a Btrfs filesystem. Analogous to the Synology Docker package, we use the @docker directory, but that could just as well be another directory of your choosing.

Instead of using the dockerd comand-line options, we can also create a file /etc/docker/daemon.json with the following content:

{
   "data-root" : "/volume1/@docker",
   "storage-driver" : "btrfs"
}Code language: JSON / JSON with Comments (json)

Start the Docker daemon with the --data-root option.

root@DS:~# dockerd --data-root /volume1/@docker --storage-driver btrfs --log-level error &
[1] 14709
root@DS:~# ERRO[2021-03-30T16:29:10.975639603+02:00] cleanup working directory in namespace        error="open /volume1/@docker/containerd/daemon/io.containerd.runtime.v2.task/moby: no such file or directory" namespace=moby

root@DS:~#Code language: Markdown (markdown)

The Docker daemon gives an error message, but seems to start.

Unfortunately, running the hello-world container gives the following error message.

root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest
ERRO[2021-03-30T16:30:30.001596712+02:00] Handler for POST /v1.41/containers/create returned error: Failed to create btrfs snapshot: inappropriate ioctl for device
docker: Error response from daemon: Failed to create btrfs snapshot: inappropriate ioctl for device.
See 'docker run --help'.
root@DS:~#Code language: Markdown (markdown)

The error is: Failed to create btrfs snapshot: inappropriate ioctl for device. This is the same error message that you will find on the internet when trying to upgrade the Docker version installed by the Synology Docker package.

The error could be caused because the version of btrfs-progs (that provides the btrfs command line tool) expects a more updated version of the btrfs kernel module.

At the time of writing no solution is available.

As an alternative, in the next section we will have a look at the storage driver vfs.

Storage driver vfs

The VFS storage driver is not a union filesystem; instead, each layer is a directory on disk, and there is no copy-on-write support. To create a new layer, a “deep copy” is done of the previous layer. This leads to lower performance and more space used on disk than other storage drivers. However, it is robust, stable, and works in every environment. It can also be used as a mechanism to verify other storage back-ends against, in a testing environment.

root@DS:~# dockerd --data-root /volume1/@docker --storage-driver vfs --log-level error &
[1] 16883
root@DS:~# ERRO[2021-03-30T20:57:08.481473625+02:00] cleanup working directory in namespace        error="open /volume1/@docker/containerd/daemon/io.containerd.runtime.v2.task/moby: no such file or directory" namespace=moby

root@DS:~#
Code language: Markdown (markdown)

The Docker daemon gives the error message we saw with the storage driver btrfs, but seems to start.

To test if everything works correctly, we will run the hello-world container that displays a welcome message.

root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu markdown

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@DS:~#Code language: Markdown (markdown)

At this point we have a working Docker installation. Because the Synology Docker GUI is not available, as an alternative, Portainer could be used as a convenient container management tool.

Conclusion

Although it seems the latest Docker version will work without problems on an ext4 filesystem on a Synology NAS, the Btrfs filesystem continues to cause problems. Synology focuses on the Btrfs filesystem, so many users will already use it or will migrate to a Btrfs filesystem in the near future. If you own a Synology with enough bays, you could consider formatting a disk with the ext4 filesystem and using it for Docker.

It turns out it’s not a matter of upgrading the Docker version and be done with it. Apparently some adjustments still need to be made so that using the btrfs storage driver also works on the Btrfs filesystem on a Synology NAS. The problems could be caused because the version of btrfs-progs expects a more updated version of the btrfs kernel module. Maybe that’s why Synology is currently reluctant to upgrade the Docker version.

As an alternative the vfs storage driver can be used on a Btrfs (or any other) filesystem. But is must be said that this is not the most optimal solution and comes with possible performance and disk space penalties.