After posting part 1 of this series, Korikaze on the VyOS Forums brought up an excellent point about installing packages on VyOS. The installation of ZeroTier will persist across reboots, but upgrading to a new image is a different story.
What Is The Issue
VyOS handles upgrades like a network device would, and not how you expect for a normal Linux Distro. When upgrading, they fully write a new OS installation. This is pretty smart on their part, because it eliminates a lot of the problems that occur with long lived OS installations with multiple installs/uninstalls. Think about the amount of times a new installation of Windows solved all of your headaches.
This does present some drawbacks. Anything you install in addition to VyOS will not persist after a new installation. So when we install and setup ZeroTier, the ZeroTier installation is blown out when we install a new VyOS image.
We do have some options to solve for this, which I’ll cover in this post.
Option 1: Creating a Symlink To The zerotier-one Folder
VyOS has a directory that will persist between upgrades, located in /config. If we put something in there, it will still be there after booting to a new image. ZeroTier’s configuration is stored in /var/lib/zerotier-one, so if we can have the configuration stored under the /config folder, then it will persist after upgrades.
We can do this with a symlink. Let’s install ZeroTier again on a fresh VyOS installation.
Setup routing to the internet and name resolution.
set interfaces ethernet eth0 address dhcp
set system name-server 4.2.2.2
Install ZeroTier (take note of the ZeroTier address here and remember it for later)
sudo su
curl -s https://install.zerotier.com | sudo bash
*** Success! You are ZeroTier address [ ebxxxxxxxx ].
Here is where this will differ from the other posts. We’ll create a symlink. First we need to move the zerotier-one folder to the /config folder. Once it’s there, we can create a symlink, so the application can still pull its configuration from where it expects it to be.
cd /var/lib
mv /var/lib/zerotier-one /config/scripts/
ln -s /config/scripts/zerotier-one
root@vyos:/var/lib# ls -la | grep zerotier
lrwxrwxrwx 1 zerotier-one zerotier-one 28 Jan 10 15:24 zerotier-one -> /config/scripts/zerotier-one
root@vyos:/var/lib# ls /config/scripts/zerotier-one/
authtoken.secret metrics.prom zerotier-idtool zerotier-one.te
controller.d metricstoken.secret zerotier-one
identity.public planet zerotier-one.pid
identity.secret zerotier-cli zerotier-one.port
Now we can create our devicemap to map the zerotier interface to an ethernet interface.
root@vyos:/var/lib# cd zerotier-one
root@vyos:/var/lib/zerotier-one# cat >devicemap
xxxxxxxxxxxxxxxx=eth10
^C
We need to restart zerotier for that change to take effect.
root@vyos:/var/lib/zerotier-one# sudo /etc/init.d/zerotier-one restart
Restarting zerotier-one (via systemctl): zerotier-one.service.
Joining the Network
root@vyos:/var/lib/zerotier-one# zerotier-cli join xxxxxxxxxxxxxxxx
200 join OK
Verification
vyos@vyos:~$ show interfaces | match eth10
eth10 10.13.70.67/16 aa:bb:cc:dd:ee:f0 default 2800 u/u
Let’s reboot just to make sure the ZeroTier deployment is persistent across normal reboots.
vyos@vyos:~$ reboot now
Broadcast message from root@debian on pts/0 (Wed 2024-01-10 15:08:51 UTC):
The system will reboot now!
Welcome to VyOS!
vyos@vyos:~$ show interfaces | match eth10
eth10 10.13.70.67/16 aa:bb:cc:dd:ee:f0 default 2800 u/u
We can see that the installation can survive reboots just fine.
Installing The New Image
Now we can work on installing a new image for an “upgrade”. We first need to add the image. Just follow the guided prompts.
Note: We’re just going to install the same image we’re already using and give it a different name. Normally you’d be upgrading to a new image (e.g. 1.4.0 to 1.4.1).
vyos@vyos:~$ add system image https://cdn.vyos.io/1.4.0-rc1/vyos-1.4.0-rc1-amd64.iso
The file is 410.000 MiB.
[#################################################################] 100%
What would you like to name this image? (Default: 1.4.0-rc1) 1.4.0-new
Would you like to set the new image as the default one for boot? [Y/n] y
An active configuration was found. Would you like to copy it to the new image? [Y/n] y
Would you like to copy SSH host keys? [Y/n] y
Verify the new image is installed and set as the boot image
vyos@vyos:~$ show system image
Name Default boot Running
--------- -------------- ---------
1.4.0-new Yes
1.4.0-rc1 Yes
Let’s reboot now to boot to the new image. You should see yourself on the new image after the reboot.
vyos@vyos:~$ reboot now
Broadcast message from root@vyos on pts/0 (Wed 2024-01-10 15:16:14 UTC):
The system will reboot now!
Welcome to VyOS!
vyos@vyos:~$ show system image
Name Default boot Running
--------- -------------- ---------
1.4.0-new Yes Yes
1.4.0-rc1
Let’s see if our ZeroTier configuration is still present.
vyos@vyos:~$ ifconfig | match eth10
root@vyos:/home/vyos# zerotier-cli info
bash: zerotier-cli: command not found
Predictably, ZeroTier is no longer installed. Let’s reinstall it.
vyos@vyos:~$ sudo su
root@vyos:/home/vyos# curl -s https://install.zerotier.com | sudo bash
*** Success! You are ZeroTier address [ 28xxxxxxxx ].
If you remember our ZeroTier address from before, you can see these are now gone. This is because our symlinks are also gone with the new image installed. We’ll need to recreate those.
root@vyos:/home/vyos# cd /var/lib
ln -s /config/scripts/zerotier-one
ln: failed to create symbolic link './zerotier-one': File exists
You can see that it fails. This is because after the new installation of ZeroTier, the /var/lib/zerotier-one folder exists. We’ll need to delete that before we can recreate the symlink.
root@vyos:/var/lib# rm -r zerotier-one/
root@vyos:/var/lib# cd /var/lib
ln -s /config/scripts/zerotier-one
We’ll need to reboot the ZeroTier service to have it pull the correct configuration.
root@vyos:/var/lib/zerotier-one# sudo /etc/init.d/zerotier-one restart
Restarting zerotier-one (via systemctl): zerotier-one.service.
Now let’s check our ZeroTier address.
root@vyos:/var/lib/zerotier-one# zerotier-cli info
200 info ebxxxxxxxx 1.12.2 ONLINE
Now we’re back to our original address, let’s see if eth10 returned.
root@vyos:/var/lib/zerotier-one# show interfaces | match eth10
eth10 10.13.70.67/16 aa:bb:cc:dd:ee:f0 default 2800 u/u
It’s back, let’s make sure we can ping another ZeroTier node.
vyos@vyos:~$ ping 10.13.0.1
PING 10.13.0.1 (10.13.0.1) 56(84) bytes of data.
64 bytes from 10.13.0.1: icmp_seq=1 ttl=64 time=175 ms
64 bytes from 10.13.0.1: icmp_seq=2 ttl=64 time=2.37 ms
So the process isn’t necessarily as streamlined as you’d want it to be, but there are a few things to make that a little easier.
Packaging ZeroTier with VyOS
If you build your own VyOS images, adding additional debian packages is as easy as dropping the .deb file into the /vyos-build/packages folder. You can find the ZeroTier packages here: https://download.zerotier.com/debian/
Remember to grab the correct package for your image:
- Debian Jessie for VyOS 1.2 (crux)
- Debian Buster for VyOS 1.3 (equuleus)
- Debian Bookworm for VyOS 1.4 (sagitta)
- Debian Bookworm for the upcoming VyOS 1.5/circinus/current (subject to change) – aka the rolling release
Recreating The Symlink With The Postconfig Script
VyOS has a script that is run after VyOS fully boots. We can do our tasks of deleting the /var/lib/zerotier-one folder and recreating our symlink in there. Here is a basic example:
#!/bin/sh
# This script is executed at boot time after VyOS configuration is fully applied.
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.
symlink_path="/var/lib/zerotier-one"
# Check if the symlink exists and points to a valid file
if [ -L "$symlink_path" ] && [ -e "$symlink_path" ]; then
echo "Symlink exists and is valid."
else
# Perform your actions here
rm -r /var/lib/zerotier-one
ln -s /config/scripts/zerotier-one /var/lib/zerotier-one
fi
By packaging ZeroTier with VyOS, and setting the postconfig script, you can make image upgrades be seamless, though the process is not as simple as you may want.
Option 2: Install ZeroTier Into a Container
Another option that VyOS provides is the addition of containers. You can install multiple services into containers, including ZeroTier. This may present a better solution for keeping ZeroTier persistent across image upgrades. It also makes managing ZeroTier versions a little easier since you don’t need to do that with Linux.
We need to add the image for the container first. This will download the image. Notice we set the image to the “latest” tag, but you can install a specific version if you wanted. The available tags can be found here: https://hub.docker.com/r/zerotier/zerotier/tags
vyos@vyos:~$ add container image zerotier/zerotier:latest
We need to create a directory in the /config folder so the container will have persistent storage. I like to have a folder dedicated for containers volume paths.
vyos@vyos:~$ sudo mkdir -p /config/containers/zt1
Now let’s configure the container in VyOS.
set container name zt1 allow-host-networks
set container name zt1 cap-add 'net-admin'
set container name zt1 cap-add 'sys-admin'
set container name zt1 device tun destination '/dev/net/tun'
set container name zt1 device tun source '/dev/net/tun'
set container name zt1 image 'zerotier/zerotier:latest'
set container name zt1 volume ZT_Path destination '/var/lib/zerotier-one'
set container name zt1 volume ZT_Path source '/config/containers/zt1'
commit
You can verify the status of the container with the below command:
vyos@vyos:~$ show container
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b82dd2bf5689 docker.io/zerotier/zerotier:latest 6 minutes ago Up 6 minutes ago zt1
If you don’t see the container in there, it is likely not running. Just run this:
vyos@vyos:~$ restart container zt1
Container "zt1" restarted!
Let’s verify that our persistent volume worked correctly. This won’t be created until the first time the container starts.
vyos@vyos:~$ sudo ls /config/containers/zt1/
authtoken.secret identity.secret networks.d zerotier-one.port
controller.d metrics.prom planet
identity.public metricstoken.secret zerotier-one.pid
Everything looks good, now we just need to create our devicemap like before and join a network.
vyos@vyos:~$ sudo su
root@vyos:/home/vyos# cd /config/containers/zt1/
root@vyos:/config/containers/zt1# cat >devicemap
xxxxxxxxxxxxxxxx=eth10
^C
Now we can access the shell of our container to join the zerotier network. Remember to authorize it in ZeroTier Central.
vyos@vyos:~$ connect container zt1
# zerotier-cli join xxxxxxxxxxxxxxxx
200 join OK
Let’s verify everything is working.
vyos@vyos:~$ show interfaces | match eth10
eth10 10.13.165.85/16 aa:bb:cc:dd:ee:f0 default 2800 u/u
Let’s see what it looks like after upgrading to a new image.
vyos@vyos:~$ add system image https://cdn.vyos.io/1.4.0-rc1/vyos-1.4.0-rc1-amd64.iso
vyos@vyos:~$ show system image
Name Default boot Running
--------- -------------- ---------
1.4.0-new Yes
1.4.0-rc1 Yes
vyos@vyos:~$ reboot now
Broadcast message from root@vyos on pts/0 (Wed 2024-01-10 15:16:14 UTC):
The system will reboot now!
Welcome to VyOS!
vyos@vyos:~$ show system image
Name Default boot Running
--------- -------------- ---------
1.4.0-new Yes Yes
1.4.0-rc1
vyos@vyos:~$ show interfaces | match eth10
eth10 10.13.165.85/16 aa:bb:cc:dd:ee:f0 default 2800 u/u
vyos@vyos:~$ show container
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3df4a182e639 docker.io/zerotier/zerotier:latest About a minute ago Up About a minute ago zt1
We can see that is much easier and more consistent than our symlink method. It just worked without any additional user intervention.
Conclusion
I gave you 2 options for making our ZeroTier deployment persistent in VyOS. I think there’s a clear winner between the 2, since containers don’t require any additional steps for a system upgrade. It also makes version management much easier since you don’t have to worry about screwing up VyOS by upgrading packages.
Video
My friend is making this series into Videos on YouTube:
Leave a Reply