Table of Contents
In this post we are going to install the latest Docker binaries on a Synology NAS without using the official Synology Docker package. Because the Docker version that is currently installed by the this package is pretty old, upgrading to a newer version is a hot topic in the Synology community.
There are scripts available on the internet for upgrading the Docker version that is installed by the Synology Docker package, but they all fail when using the Btrfs file system. To get to the heart of the problem, I will install the Docker binaries without using the Synology Docker package on a clean installation of DSM.
For this test I am using a vanilla installation of DSM 7.0-41222 Beta on VirtualDSM and the latest Docker binaries from the Docker website. At the time of writing the latest Docker version is 20.10.5 (2021-03-02) and the Docker version installed by the Synology Docker package is 18.09.8 (2019-07-17).
We are going to follow the instructions in the Install Docker Engine from binaries page and because the Synology NAS is based on Linux, we move directly to the section Install daemon and client binaries on Linux.
Please note that the commands executed below can irreparably damage your Synology NAS and cause data to be permanently lost. So be very careful when trying it out yourself.
Prerequisites
Since Synology is also releasing a working Docker package, as expected, we meet the prerequisites set in the instructions.
root@DS:~# uname -a
Linux DS 4.4.180+ #41222 SMP Fri Dec 4 19:02:11 CST 2020 x86_64 GNU/Linux synology_kvmx64_virtualdsm
root@DS:~# iptables -V
iptables v1.8.3 (legacy)
root@DS:~# ps -V
ps from procps-ng 3.3.15
root@DS:~# xz -V
xz (XZ Utils) 5.2.1
liblzma 5.2.1
root@DS:~# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 7 1 1
cpu 4 1 1
cpuacct 6 1 1
blkio 8 1 1
memory 5 1 1
devices 3 1 1
freezer 9 1 1
root@DS:~#
Code language: Markdown (markdown)
Install Docker daemon and client binaries
Download the static binary archive. Go to https://download.docker.com/linux/static/stable/ (or change stable
to nightly
or test
), choose your hardware platform, and download the .tgz
file relating to the version of Docker Engine you want to install.
In this case we are using the latest stable version 20.10.5 for hardware platform x86_64.
root@DS:~# wget -q https://download.docker.com/linux/static/stable/x86_64/docker-20.10.5.tgz
root@DS:~# ls
docker-20.10.5.tgz
root@DS:~#
Code language: Markdown (markdown)
Extract the archive using the tar
utility. The dockerd
and docker
binaries are extracted in a directory docker
.
root@DS:~# tar xvzf docker-20.10.5.tgz
docker/
docker/docker-init
docker/docker
docker/containerd-shim
docker/ctr
docker/docker-proxy
docker/runc
docker/containerd-shim-runc-v2
docker/containerd
docker/dockerd
root@DS:~#
Code language: Markdown (markdown)
For now we are not going to move the binaries to /usr/local/bin
as described in the instructions, but we are going to adjust the PATH
variable so the binaries can be found when we want to execute them.
root@DS~# PATH=/root/docker:$PATH
root@DS:~# echo $PATH
/root/docker:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin
root@DS:~#
Code language: Markdown (markdown)
Because by default we use the aufs
storage driver, I guess we also need the auplink
binary. For now I will copy this binary from an existing Synology NAS with the Synology Docker package installed.
root@DS:~# scp sysop@DS716:/usr/local/bin/auplink docker/.
admin@192.168.0.111's password:
auplink 100% 14KB 9.8MB/s 00:00
root@DS:~#
Code language: Markdown (markdown)
Start Docker daemon
At this moment, we only care about errors. That is why we increase the logging level to “error” before starting the Docker daemon.
root@DS:~# dockerd --log-level error
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.3 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
root@DS:~#
Code language: Markdown (markdown)
The error message indicates that we perhaps need to upgrade iptables or the kernel. If we take a closer look at the messages, we probably are missing some kernel modules.
Insert kernel modules
The code below is from the start-stop-status
script that is provided by the official Synology Docker package. Let’s see if this will solve the problem and execute the code. This code will insert the necessary modules into the kernel.
source /usr/syno/etc.defaults/iptables_modules_list
DockerModules="xt_addrtype.ko xt_conntrack.ko veth.ko macvlan.ko aufs.ko"
DockerBridgeModules="llc.ko stp.ko bridge.ko macvlan.ko"
if [ -f /lib/modules/br_netfilter.ko ]; then
DockerBridgeModules="${DockerBridgeModules} br_netfilter.ko"
fi
DockerIngressModules="iptable_mangle.ko xt_mark.ko ip_vs.ko ip_vs_rr.ko xt_ipvs.ko"
DockerServName="docker"
InsertModules="${KERNEL_MODULES_CORE} ${KERNEL_MODULES_COMMON} ${KERNEL_MODULES_NAT} ${IPV6_MODULES} ${DockerModules} ${DockerBridgeModules}"
if [ -f /lib/modules/ip_vs.ko -a -f /lib/modules/ip_vs_rr.ko -a -f /lib/modules/xt_ipvs.ko ]; then
InsertModules="${InsertModules} ${DockerIngressModules}"
fi
iptablestool --insmod "${DockerServName}" ${InsertModules}
Code language: Bash (bash)
let’s try to start the docker daemon again. Using the ampersand (&) will start the process in the background.
root@DS:~# dockerd --log-level error &
[1] 13924
root@DS:~#
Code language: Markdown (markdown)
Run hello-world
Now the Docker daemon does start without errors. To test if everything works correctly, we will run the hello-world container that displays a welcome message.
root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu markdown
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
root@DS:~#
Code language: Markdown (markdown)
At this point we have a working Docker installation. Because the Synology Docker GUI is not available, as an alternative, Portainer could be used as a convenient container management tool.
Although the Docker installation works, the Docker files are stored on the root file system where the available space is limited. Also, the storage driver aufs
is deprecated as we will see in the next section.
Storage drivers
Storage driver aufs
AUFS is a union filesystem. The aufs
storage driver was previously the default storage driver used for managing images and layers on Docker for Ubuntu, and for Debian versions prior to Stretch. If your Linux kernel is version 4.0 or higher, and you use Docker Engine – Community, consider using the newer overlay2, which has potential performance advantages over the aufs
storage driver.
Let’s have a closer look at the docker info output.
root@DS:~# docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.5
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
apparmor
Kernel Version: 4.4.180+
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859GiB
Name: DS
ID: PO3D:IIPE:M6KW:TFPS:6JPY:DP4T:ZUOY:AVGP:KCEC:6CFG:OQLM:KFE3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.
root@DS:~#
Code language: Markdown (markdown)
As you can see the storage driver is aufs
, and there is also a warning at the bottom that the aufs
storage-driver is deprecated, and will be removed in a future release.
Storage driver overlay2
OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. Docker provides two storage drivers for OverlayFS: the original overlay
, and the newer and more stable overlay2
.
First, let’s stop the Docker daemon and try another storage driver.
root@DS:~# ps -ef | grep dockerd
root 13924 12415 8 16:14 pts/1 00:00:00 dockerd --log-level error
root 14167 12415 0 16:14 pts/1 00:00:00 grep --color=auto dockerd
root@DS:~# kill 13924
root@DS:~#
Code language: Markdown (markdown)
Docker supports a number of storage drivers. Because overlay2
was suggested earlier as an alternative to aufs
, let’s give that a try.
root@DS:~# dockerd --storage-driver overlay2 --log-level error
ERRO[2021-03-30T16:20:45.969233496+02:00] failed to mount overlay: no such device storage-driver=overlay2
failed to start daemon: error initializing graphdriver: driver not supported
root@DS:~#
Code language: Markdown (markdown)
Using this driver does not appear to work. Furthermore, the Docker files are stored on the root filesystem which is limited in space. It is better to store the Docker files on /volume1
. Because the file system on /volume1
is Btrfs, let’s focus on the storage driver btrfs
.
Storage driver btrfs
Btrfs is a next generation copy-on-write filesystem that supports many advanced storage technologies that make it a good fit for Docker. Btrfs is included in the mainline Linux kernel.
Docker’s btrfs
storage driver leverages many Btrfs features for image and container management. Among these features are block-level operations, thin provisioning, copy-on-write snapshots, and ease of administration. You can easily combine multiple physical block devices into a single Btrfs filesystem.
Start the Docker deamon with storage driver btrfs
.
root@DS:~# dockerd --storage-driver btrfs --log-level error
failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)
root@DS:~#
Code language: Markdown (markdown)
This actually makes sense because the root filesystem is ext4 and not Btrfs. In other words: we are using the btrfs
storage driver on the wrong filesystem.
Let’s point the data root to /volume1/@docker
which is located on a Btrfs filesystem. Analogous to the Synology Docker package, we use the @docker
directory, but that could just as well be another directory of your choosing.
Instead of using the dockerd comand-line options, we can also create a file /etc/docker/daemon.json
with the following content:
{
"data-root" : "/volume1/@docker",
"storage-driver" : "btrfs"
}
Code language: JSON / JSON with Comments (json)
Start the Docker daemon with the --data-root
option.
root@DS:~# dockerd --data-root /volume1/@docker --storage-driver btrfs --log-level error &
[1] 14709
root@DS:~# ERRO[2021-03-30T16:29:10.975639603+02:00] cleanup working directory in namespace error="open /volume1/@docker/containerd/daemon/io.containerd.runtime.v2.task/moby: no such file or directory" namespace=moby
root@DS:~#
Code language: Markdown (markdown)
The Docker daemon gives an error message, but seems to start.
Unfortunately, running the hello-world container gives the following error message.
root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest
ERRO[2021-03-30T16:30:30.001596712+02:00] Handler for POST /v1.41/containers/create returned error: Failed to create btrfs snapshot: inappropriate ioctl for device
docker: Error response from daemon: Failed to create btrfs snapshot: inappropriate ioctl for device.
See 'docker run --help'.
root@DS:~#
Code language: Markdown (markdown)
The error is: Failed to create btrfs snapshot: inappropriate ioctl for device. This is the same error message that you will find on the internet when trying to upgrade the Docker version installed by the Synology Docker package.
The error could be caused because the version of btrfs-progs
(that provides the btrfs
command line tool) expects a more updated version of the btrfs
kernel module.
At the time of writing no solution is available.
As an alternative, in the next section we will have a look at the storage driver vfs
.
Storage driver vfs
The VFS storage driver is not a union filesystem; instead, each layer is a directory on disk, and there is no copy-on-write support. To create a new layer, a “deep copy” is done of the previous layer. This leads to lower performance and more space used on disk than other storage drivers. However, it is robust, stable, and works in every environment. It can also be used as a mechanism to verify other storage back-ends against, in a testing environment.
root@DS:~# dockerd --data-root /volume1/@docker --storage-driver vfs --log-level error &
[1] 16883
root@DS:~# ERRO[2021-03-30T20:57:08.481473625+02:00] cleanup working directory in namespace error="open /volume1/@docker/containerd/daemon/io.containerd.runtime.v2.task/moby: no such file or directory" namespace=moby
root@DS:~#
Code language: Markdown (markdown)
The Docker daemon gives the error message we saw with the storage driver btrfs
, but seems to start.
To test if everything works correctly, we will run the hello-world container that displays a welcome message.
root@DS:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu markdown
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
root@DS:~#
Code language: Markdown (markdown)
At this point we have a working Docker installation. Because the Synology Docker GUI is not available, as an alternative, Portainer could be used as a convenient container management tool.
Conclusion
Although it seems the latest Docker version will work without problems on an ext4 filesystem on a Synology NAS, the Btrfs filesystem continues to cause problems. Synology focuses on the Btrfs filesystem, so many users will already use it or will migrate to a Btrfs filesystem in the near future. If you own a Synology with enough bays, you could consider formatting a disk with the ext4 filesystem and using it for Docker.
It turns out it’s not a matter of upgrading the Docker version and be done with it. Apparently some adjustments still need to be made so that using the btrfs
storage driver also works on the Btrfs filesystem on a Synology NAS. The problems could be caused because the version of btrfs-progs
expects a more updated version of the btrfs
kernel module. Maybe that’s why Synology is currently reluctant to upgrade the Docker version.
As an alternative the vfs
storage driver can be used on a Btrfs (or any other) filesystem. But is must be said that this is not the most optimal solution and comes with possible performance and disk space penalties.