Andrew Katz

Running Syncthing on Fly.io with Tailscale

As a new-ish person to using services like Fly.io, the prospect of being able to send dockerized apps that once sat on a bulky server next to my desk into the cloud is still fresh and impressive. I've been sitting around thinking of innovative ways to apply this service and make my life easier.

One thing that recently came mind was Syncthing, a continuous file synchronization program that synchronizes files between two or more computers in real time.

My next thought was that adding Tailscale into the mix would be a really nice addition and match the simplicity of setting up apps on Fly. This would ensure that all connections made to our Syncthing instance on Fly.io's server are through a VPN only - not through the public internet.

From a compute aspect, you can run Syncthing for free on Fly. In terms of free-tier storage, Fly will give you 3 GB of persistent volume storage, but I would assume most people want more than this. I'm going with 15 GB which costs $1.80/mo. For networking, Fly does not charge for inbound data transfer and the free tier gives you 100GB/mo outbound free in NA and EU. If you're curious about pricing, check out this page.

If you have not used fly.io before, I would recommend following their hands-on guide. This will give you the easy steps to install Fly's CLI tool (called flyctl), sign-up for an account, and get authenticated. If you haven't used Tailscale as well, check out Tailscale quickstart. Both products are intended be highly straightforward.

This guide assumes you have already done the preliminary steps to set up these products and have basic understanding of them. I hope it inspires you to think of some new, clever ways to use these services together. If you want to reach out and share any similar projects, I am on Twitter at @andrewckatz or you can reach me at andrew[at]akatz.org


Guide

To start, clone syncthing's GitHub repository to your local machine:

git clone https://github.com/syncthing/syncthing

Open /syncthing/ as a folder in your preferred text editor. Syncthing's Dockerfile under /syncthing/Dockerfile uses multi-stage builds, so we need to take that into account when adding a stage for Tailscale. Luckily, Tailscale's documentation for Fly.io gives us the almost the exact code we need.

As Tailscale's guide says, go into Tailscale's Admin console under Settings > Keys, and create an ephemeral authkey.

Sidenote: I recommend not using a reusable key. Your Fly machine shouldn't die often, and if it does, then just generate a new key and set the new secret using flyctl. With a reusable key, if your machine fails for whatever reason and restarts over and over, you may end up with 20+ machines on Tailscale which is a pain to clear out.

Image title

Your new key should start with tskey and you can just copy it over to a text editor for now. Eventually, we will be setting it to the environment variable $TAILSCALE_AUTHKEY and storing it securely within Fly itself.

Now, we are going to configure Syncthing's Dockerfile to install Tailscale. I'll explain the changes that I'm making and just give you the whole Dockerfile at the end just to alleviate any possible missteps.

First, under the first build step called builder, we'll insert the Tailscale build step. Tailscale uses 'app' as the working directory value for their example, but I'm changing this to 'src' since that is what Syncthing uses. I don't think this makes a big difference either way.

Second, in the final build step, we are adding some packages related to certificates and iptables and just trying to keep the size of the package down.

The third thing we will do is copying over Tailscale's binaries to the final production image. Your Dockerfile should look like this when all is said and done:

ARG GOVERSION=latest
FROM golang:$GOVERSION AS builder

WORKDIR /src
COPY . .

ENV CGO_ENABLED=0
ENV BUILD_HOST=syncthing.net
ENV BUILD_USER=docker
RUN rm -f syncthing && go run build.go -no-upgrade build syncthing

FROM alpine:latest as tailscale
WORKDIR /src
COPY . ./
ENV TSFILE=tailscale_1.30.1_amd64.tgz
RUN wget https://pkgs.tailscale.com/stable/${TSFILE} && tar xzf ${TSFILE} --strip-components=1
COPY . ./


FROM alpine

RUN apk update && apk add ca-certificates iptables ip6tables && rm -rf /var/cache/apk/*

EXPOSE 8384 22000/tcp 22000/udp 21027/udp

VOLUME ["/var/syncthing"]

RUN apk add --no-cache ca-certificates su-exec tzdata

COPY --from=builder /src/syncthing /bin/syncthing
COPY --from=builder /src/script/docker-entrypoint.sh /bin/entrypoint.sh
COPY --from=tailscale /src/tailscaled /src/tailscaled
COPY --from=tailscale /src/tailscale /src/tailscale
RUN mkdir -p /var/run/tailscale /var/cache/tailscale /var/lib/tailscale

ENV PUID=1000 PGID=1000 HOME=/var/syncthing

HEALTHCHECK --interval=1m --timeout=10s \
  CMD nc -z 127.0.0.1 8384 || exit 1

ENV STGUIADDRESS=0.0.0.0:8384
ENTRYPOINT ["/bin/entrypoint.sh", "/bin/syncthing", "-home", "/var/syncthing/config"]

The next thing that we have to do is modify the entrypoint script located in /script/docker-entrypoint.sh. These additions make sure that we are starting up Tailscale and getting authorized when we run the container. Your entry point script should look like this at the end:

#!/bin/sh

/src/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
/src/tailscale up --authkey=${TAILSCALE_AUTHKEY} --hostname=fly-app

set -eu

if [ "$(id -u)" = '0' ]; then
  chown "${PUID}:${PGID}" "${HOME}" \
    && exec su-exec "${PUID}:${PGID}" \
       env HOME="$HOME" "[email protected]"
else
  exec "[email protected]"
fi

Sidenote: Under --hostname, instead of fly-app, you can put whatever name you want to be listed in Tailscale.

Now we're ready to start configuring from flyctl. In your terminal, run fly launch to start setting up your app environment. Choose your app name, where you will run the container (keep in mind there are different costs associated with different locations - see Fly's pricing for details), and say N to setting up a Postgres database and to deploying.

ak@mac syncthing % fly launch
Creating app in /Users/ak/PycharmProjects/syncthing
Scanning source code
Detected a Dockerfile app
? App Name (leave blank to use an auto-generated name): 
Automatically selected personal organization: notmyrealemail@theranos.com
? Select region:  [Use arrows to move, type to filter]
  dfw (Dallas, Texas (US))
  ewr (Secaucus, NJ (US))
  fra (Frankfurt, Germany)
  gru (São Paulo)
  hkg (Hong Kong, Hong Kong)
? Select region: lax (Los Angeles, California (US))
Created app crimson-pine-8091 in organization personal
Wrote config file fly.toml
? Would you like to set up a Postgresql database now? No
? Would you like to deploy now? No
Your app is ready. Deploy with `flyctl deploy`

We need to set up persistent volume storage at this point which is also done through flyctl. You can tailor this command to your needs (name, size, and region), but the basic syntax is: flyctl volumes create <name> --region <region> --size <number>.

I'm going to go with flyctl volumes create syncstore --region lax --size 15

ak@mac syncthing % flyctl volumes create syncstore --region lax --size 15 
        ID: vol_6d7xkrk53jyrw2q9
      Name: syncstore
       App: crimson-pine-8091
    Region: lax
      Zone: c6d5
   Size GB: 15
 Encrypted: true
Created at: 11 Sep 22 22:00 UTC

In our Dockerfile, Syncthing wants to mount a volume called /var/syncthing. The way that we our app about this is by editing the /syncthing/fly.toml file which gets automatically generated after you run fly launch. We are going to add a [[mount]] section to this file and put the information about our volume in that section. The source is the name of the fly volume.

The other aspect that we need to change about the fly.toml file is that we're going to not expose any ports/services to the outside world. The Docker container itself will expose these so that they are accessible to Tailscale, but we don't want them to accessible to the public internet as well. The way that we do this is by removing everything and under and including the line [[services]].

After doing this, your fly.toml should look something like this (depending on how you named your volume):

# fly.toml file generated for crimson-pine-8091 on 2022-09-11T14:51:58-07:00

app = "crimson-pine-8091"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[mounts]
source="syncstore"
destination="/var/syncthing"

Remember the TAILSCALE_AUTHKEY we created? It's time to set that using flyctl which interacts with Fly's built-in secret store. You can add your authkey using the following command: flyctl secrets set TAILSCALE_AUTHKEY=tskey-<key>

ak@mac syncthing % flyctl secrets set TAILSCALE_AUTHKEY=tskey-123456790420fasdf
Secrets are staged for the first deployment%

Fly responds with the output that our secrets are staged for the first deployment.

Protip: If you ever want to check in the future what secrets you have staged on a project, you can type in fly secrets list. Fly just stores a hash of this secret:

ak@mac syncthing % fly secrets list
NAME                    DIGEST                  CREATED AT 
TAILSCALE_AUTHKEY       662afb463edc7ae1        1m41s ago

At this point, we are ready to deploy - and you can do that by typing fly deploy.

As the build process happens, I like to open the Fly dashboard and check out the Monitoring section. After a minute or two, your Syncthing instance should be available on Tailscale's Machines page.

Image title

I recommend saving yourself a headache and setting this key to not expire by clicking on the 3 dots next to the machine. Click 'Disable key expiry'.

You should be able to access your Syncthing over Tailscale over port 8483 at this point (e.g., connect to https://100.108.5.238:8384). You should NOT be able to access it using the FQDN (e.g., https://crimson-pine-8091.fly.dev:8384) If you're using Magic DNS, you can also just use the Tailscale hostname of the machine that we set up. If you're not using Magic DNS, why not check it out? It's pretty badass.

Important

You may have an issue getting discovery to work initially. I've found that if I manually set the opposite machine's Tailscale IP with port 22000 under the Advanced tab of Edit Device on both the local and remote machines, then they will find each other:

Image title

Other considerations

One thing that you will notice is that Syncthing uses a self-signed certificate, so you will see 'This page is not secure' when visiting it. I haven't tried this yet, but another cool thing to consider is Tailscale's TLS features to generate SSL certs that can be used over your Tailnet. For more information, check out this page: Enabling HTTPS · Tailscale. I would be interested to know if anyone has got this working.

So far, the biggest caveat that I am dealing with in this set up is the throughput (~1MB up/down). However, time is not a huge factor in how I am using Syncthing - as long as the files get to where they're going eventually, it's fine with me.

Image title

Another cool thing is that Fly now has a built-in Grafana feature, so you can see your network IO, volume usage, etc. in real time:

Image title

Image title

Thanks for reading!

- 0 toasts