My Journey to SyncThing

SyncThing is an open source, cross platform, decentralized, secure, folder sync solution. In short, it allows you to sync complete folders peer-to-peer between your computers with ease. The setup is easy and does not require any server to be available 24/7 (hence decentralized). Moreover, you DO NOT need any internet connection once you install it on your computers (unlike other solutions).

The Journey Begins Here –

In order to fully understand my necessity in SyncThing, I’ll take a step back and explain. My typical development environment in the organizations I work for includes a Windows machine and another Ubuntu machine. While I develop on Ubuntu, the organizations heavily use Microsoft and Windows-based products. I find emulating or finding a substitute for them on my development machine a hassle.

There are certain files I need to share between both machines – from code, to documents to configurations. Lately I also had to compile the code from my Windows machine on the Linux machine. I was looking for a way that will allow me to benefit from both machines by sharing the files one way or another.

Source Control

You’re probably thinking – “Use Git or some other proper source control tools! The tools will do the work for you!”

At first, I thought so too, but there were too many projects and too many scripts to start organizing everything into repositories and pushing all in. I had binaries and libraries in there as well. I didn’t need any versioning solution between the machines, just a way for me to work as though they are one. Some projects are controlled in Git, others in TFS and the rest are either on some network drive or in the JFrog Artifactory. People are using them and they all have a reason to be where they are at this point in time. I wasn’t looking to change my surroundings or halt anyone’s workflow – I was looking for a local solution.

Mounting / Shared Folders

It seemed like a nice way to share files: all files were immediately available on both machines. I could edit documents on both machines, read and move files all through the mounted drive. I was a happy camper!

\* a few moments later *\

I was given a task to compile and run projects I worked on the Windows machine on my Linux – or run some script I developed on my Ubuntu machine on my Windows machine.

Compiling remote code takes forever to compile. It was a nightmare. Moreover I didn’t need some of the files to be propagated in the mounted drive and I definitely didn’t want to change anything in the compilation scripts regarding the build’s output.

Heard of rsync or unison?

Rsync

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a lqquick checkrq algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

Unison

Unison is a file-synchronization tool for OSX, Unix, and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, Subversion, BitKeeper, etc.), distributed filesystems (Coda, etc.), uni-directional mirroring utilities (rsync, etc.), and other synchronizers (Intellisync, Reconcile, etc). However, there are several points where it differs:Unison runs on both Windows and many flavors of Unix (Solaris, Linux, OS X, etc.) systems. Moreover, Unison works across platforms, allowing you to synchronize a Windows laptop with a Unix server, for example.
Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.
Unlike a distributed filesystem, Unison is a user-level program: there is no need to modify the kernel or to have superuser privileges on either host.
Unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.

What about them? Well, what if I use either tools to sync chosen directories and files from the mounted / shared folder to a local directory on my Ubuntu machine? Then, the local compilation will work like a charm and only chosen files will be synced!

First, I was looking for a bi-directional solution, so rsync had a major disadvantage. Running two rsync simultaneous processes hasn’t proven to work very well.

Unison on the other hand was designed to sync files this way. I played with it a little and it worked pretty well, up until I had to start renaming and moving files.

Apparently Unison does not work well when you move and rename files. Someone even wrote a workaround script that can be found here. As much as sometimes I accept workarounds to a problem, this time it was unacceptable. I simply could not believe there isn’t an open solution to this common problem; so I kept looking.

SyncThing

There it was. A simple site with all I needed.

Available for all Linux architectures and distributions I work with, Windows, and has a very friendly cross-platform GUI (psst check out SyncTrayzor for Windows). It syncs my folders from Windows to my Ubuntu (& vice versa) in a 10 second interval and does a wonderful job when I re-organize the files within the folders.

As I mentioned earlier, I had to compile certain projects on my Ubuntu while the code remained managed on my Windows – that was possible with this utility. Moreover, I could add exclusion arguments so the build output on my Ubuntu wouldn’t transfer back to the Windows machine. So far it works great and new versions are being released on a regular basis! Moreover, if tomorrow I start working on other machines, it will be so simple to synchronize the directory with the new machines (n-way synchronization support).

Here is a comparison table between rsync, Unison & SyncThing:

It’s important to take note that SyncThing may be slower than other alternatives in certain scenarios.

I haven’t quite dove into the security aspect of SyncThing due to my personal usage with it, but their claims are quite an interesting read.

Have a look, you won’t regret it!

References:

• Syncthing extremely slower than unison
• Syncthing Vs rsync Vs unison?
• Can you please convince me that Syncthing is safe to use?
• Detecting moved or renamed files using inodes
• Comparison of file synchronization software

Also published on Medium.