Skip to content

Instantly share code, notes, and snippets.

@Tho85
Last active August 7, 2024 12:55
Show Gist options
  • Save Tho85/6045429 to your computer and use it in GitHub Desktop.
Save Tho85/6045429 to your computer and use it in GitHub Desktop.

Build your own private, encrypted, open-source Dropbox-esque sync folder

Prerequisites:

  • One or more clients running a UNIX-like OS. Examples are given for Ubuntu 12.04 LTS, although all software components are available for other platforms as well (e.g. OS X). YMMV
  • A cheap Ubuntu 12.04 VPS with storage. I recommend Backupsy, they offer 250GB storage for $5/month. Ask Google for coupon codes.

Software components used:

  • Unison for file synchronization
  • EncFS for folder encryption

Set up file synchronization with unison

On your server:

  • Install Unison:

    $ sudo apt-get install unison
  • Add a user for our project and give him a decent password:

    $ sudo adduser encbox

On your client(s):

  • Install Unison:

    $ sudo apt-get install unison
  • Enable key-based authentication for SSH (replace your.vps.com with your VPS' hostname):

    $ ssh-keygen
    $ cat ~/.ssh/id_rsa.pub | ssh encbox@your.vps.com "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"
  • Create the folder holding our encrypted files:

    $ mkdir ~/.encbox
  • Create a unison profile in ~/.unison/encbox.prf (replace your.vps.com with your VPS' hostname):

    $ mkdir ~/.unison; tee ~/.unison/encbox.prf <<__EOF
    root = /home/$USER/.encbox
    root = ssh://encbox@your.vps.com/encbox
    prefer = ssh://encbox@your.vps.com/encbox
    
    backups = true
    times = true
    terse = true
    
    repeat = 2
    __EOF
  • Create an upstart file to start synchronization automatically (in /etc/init/encbox.conf):

    $ sudo tee /etc/init/encbox.conf <<__EOF
    description "encbox"
    
    start on desktop-session-start
    stop on desktop-shutdown
    
    setuid $USER
    setgid $USER
    env HOME=$HOME
    
    respawn
    
    script
        bash -l -c "unison encbox; sleep 2"
    end script
    __EOF
  • Start Unison:

    # either run in foreground:
    $ unison encbox
    
    # or run it as a system service:
    $ sudo service encbox start # starts your Encbox
    $ sudo service encbox stop  # stops your Encbox

    If unison complains about archive files on your client, run it once with the -ignorearchives flag:

    $ unison -ignorearchives encbox

Create an encrypted folder using EncFS

On your client(s):

  • Install EncFS:

    $ sudo apt-get install encfs
  • Mount encrypted directory:

    $ encfs ~/.encbox ~/Encbox

    Agree to have your target directory created, then choose p to use EncFS' preconfigured paranoia mode. Give your folder a decent password.

  • That's it! All files in ~/Encbox will be encrypted and synced securely between your VPS and all clients.

Tips

  • Polling:

    Unfortunately, Unison doesn't support file system notifications (i.e. inotify, libnotify), so it polls your file system for changes every 2 seconds (if you set repeat = 2 in your profile as we did above). This isn't nice, but I did not observe any negative effects on my notebook's battery life. Feel free to set the repeat parameter to some higher value if you observe negative effects. Bandwith usage for polling is negligible as well (~0.9 kB/poll, ~3.5 kbit/s).

  • Unison version

    Rumor has it you should use the same Unison version on your server as well as on all your clients, or you will run into problems. I didn't test it, though.

  • Android / iOS app:

    This setup is compatible with Boxcryptor Classic's smartphone apps. You just have to create an EncFS folder in expert mode (x) and follow the instructions at Boxcryptor's Support Desk. Then install a WebDAV server of your choice and expose /home/encbox/encbox via WebDAV (not in this document's scope).

@arunkp
Copy link

arunkp commented Jul 20, 2013

I believe it could also be improved greatly to an extent where one could serve a small local group of people as a service.

@eisensheng
Copy link

While this might be a good advice, the problem with this setup as far as i can tell is that the EncFS volume will always stay to be visible. So if under some strange circumstances someone might gain access to the system (at this point you already have lost the game) he'll be able to see all the files regardless of encryption used or not.

@j0lvera
Copy link

j0lvera commented Jul 20, 2013

@jeregrine
Copy link

@thinkxl Sync isn't open source.

@j0lvera
Copy link

j0lvera commented Jul 20, 2013

@jeregrine my bad, you're right

@Zash
Copy link

Zash commented Jul 20, 2013

Unison is nice, I've been using it for syncing (altho manually) for years. I would suggest also looking at lsyncd.

@reening
Copy link

reening commented Jul 20, 2013

As you're talking about a Dropbox replacement, adding a web client would probably be interesting too. I've been using h5ai myself and found it to be a pretty good one. It doesn't have authentication, but HTTP auth could fix that.

Copy link

ghost commented Jul 20, 2013

EncFS needs to be done at the origin before you synchronize them to the remote storage.

@akbar501
Copy link

ownCloud is an open source file sync solution that may be worth checking out:

http://owncloud.org

@gleicon
Copy link

gleicon commented Jul 20, 2013

Thanks for the how-to. I've had good experiences with http://sparkleshare.org/ so probably a mashup between them would be great.

@muhgatus
Copy link

Perhaps is http://labs.bittorrent.com/experiments/sync.html an alternative protocol for this. There are clients for android, windows, mac and iphone(soon) available. And btsync seems to be at least open concept. I had a quick look on the data which is generated by it's network protocol. It syncs the files nearly in real time, but to do this, there is a need for much network traffic. The battery of your mobile phone will go down within hours. I do not know if this is caused by the protocol it self and cannot be changed or if this is a kind of bug, because the project is not that old.

What is realy nice about this idea is that there is no real master. But this leads to the problem of conflicts, if two devices lose the sync to the network and start to modify the same file at the same time. How to decide which change will survive.

My guess would be to have something like git oder mercurial to handle conflicts. But as a matter of fact, this is not compatible with non programming people. I'd like to have a solution which is able to handle the sync of a shopping list other at least 4 devices.

Are there libraries or tools which are able to merge changes on images, word documents and mp3s for example?

Multiple changes to images may be stored as layers, so a human can handle those conflicts. Word may be as simple as a source code. Mp3s are a little bit complicated to merge. In most use cases the sync of the meta data would be the solution, but if the file is cutted or normalized this change must be synced to all other devices in the sync network. If a mp3 get's corrupted, because the sd card in the mobile phone has an error, is an other problem. Because how can a simple sync tool can decide that this change has not to be synced.

@akbar501 owncloud is nice for contacts and calendar sync, but webdav is not a realy good protocol to sync files over multiple devices. There is no good way to handle conflicts.

@emestee
Copy link

emestee commented Jul 20, 2013

Just a quick tip.

$ cat ~/.ssh/id_rsa.pub | ssh encbox@your.vps.com "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"

You can do the same with

$ ssh-copy-id encbox@your.vps.com

...and it's actually more reliable. For one, you won't wreck your authorized_keys file if you make a typo.

Moreover, latest SSH versions support signed keys. This may be the case where a separate key is warranted, but you might want to configure your box to accept any key signed by your CA rather than manage them individually.

@priyadarshan
Copy link

It seems unison is "is no longer under active development". http://en.wikipedia.org/wiki/Unison_(file_synchronizer)

Copy link

ghost commented Jul 21, 2013

I'm using Seafile in production as a replacement for Dropbox. Supports repository-level encryption of data at rest, open source and a user-friendly version management GUI. Clients for Windows / Mac / Linux and Android / iOS.

@reyman
Copy link

reyman commented Jul 21, 2013

There is also git annex :)
http://git-annex.branchable.com/

@TobiX
Copy link

TobiX commented Jul 21, 2013

Isn't this what git-annex was designed to do? With the "new" git-annex assistant, it's pretty user-friendly, too.

@gltiich
Copy link

gltiich commented Jul 22, 2013

I created a DropBox clone not long ago, used to have the instructions online but not anymore. I mostly relied on lsyncd which supports inotify and is open source;

https://code.google.com/p/lsyncd/

My goal was to make it cross-platform without writing any Windows services, so I created a 'mini vm' which mounted a windows directory via SMB, then that VM ran lsyncd and vola, inotify against a Windows directory. It actually worked but was way too crude, lol.

Unison is a bad way to go, wasteful and inelegant.

There just will not be a good DropBox clone until someone decides to write cross-platform services. Until that day you probably won't do better than a solution built on lsyncd.

There's also incrond, probably not a great option because it'd require a lot of scripting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment