meshed-PI network to syncronize folders over the internet


9 posts
by ladidaaa » Tue Jan 22, 2013 8:31 am
Hi,
I have a nice idea for Raspberry PIs:
It would be perfect if you could create a mesh-network with many RPIs that syncronize folders over the internet. So you could create a mesh network with friends, family and colleagues and have a folder that is automatically synced between all of them over the internet in a P2P or Torrent way.
So if a new file is added the PIs sync between each other and not like a centralized server which has to upload the file to each and every RPI.
And if one RPI is offline the mesh network does not fail.
If you have a large mesh-network it could also be possible to stream files from all of the mesh-PIs with a decent speed, because you can use the upload speed of many connections at once.

Something like the "Transporter" on Kickstarter, but not so closed and over-priced:
http://www.kickstarter.com/projects/transporterguy/transporter-a-new-way-to-share-access-and-protect
or aeroFS if it will ever arrive (http://www.aerofs.com)
or cubby's "unlimited direct sync" feature: https://www.cubby.com/features/

Is there a program that allows something like that?
What do you think about that?
Can someone create a project which i would like to support?
Posts: 11
Joined: Tue Jan 22, 2013 8:11 am
by Spid » Tue Jan 22, 2013 10:47 am
I will always assume you are running Raspbian for desktop and Raspbmc for XMBC !
Posts: 525
Joined: Sat Jan 19, 2013 12:47 pm
by rurwin » Tue Jan 22, 2013 12:03 pm
At first glance, rsync should do it, but it would need some extra intelligence to handle the multiple sources. Files changed differently by different people would either lose changes or need human intervention.

It should be possible to get git to do it, but you would have to write some interesting scripts.
User avatar
Moderator
Moderator
Posts: 2890
Joined: Mon Jan 09, 2012 3:16 pm
by ladidaaa » Wed Jan 23, 2013 7:32 am
@Spid:
I have googled before i posted and already found the links you posted.
I suppose I used "meshed" in the wrong context - sorry for that.
I do not want to create a (wireless)-mesh-network -- the idea is that multiple RPIs sync Folders over the internet - once they get access to it.
A bit like dropbox, but without a central "Dropbox-Server".
More like Dropbox in a torrent-style

@rurwin:
I suppose that is the way to go right now, until someone with good programming skills develops the software i mentioned.
I already use rsync to sync between my desktop and a remote server - but big files are problematic if they change on both sides.
Rsync only syncs in one direction and the files have to be completely transferred before a sync in the other direction is done, otherwise the half transferred file will replace the original file.
The most secure way with rsync would be to have a main-server which manages the folder with humen interaction, and then syncs the changes to the others. But they do not have the right to change the folder.
I suppose my problem is too special but i hope one day someone copies "the transporter" to open-source.
Thanks a lot for your answers!
Posts: 11
Joined: Tue Jan 22, 2013 8:11 am
by Thijxx » Wed Jan 23, 2013 10:24 am
Maybe you can use OwnCloud?
OwnCloud & Raspberry: http://www.instructables.com/id/Raspber ... box-clone/

OwnCloud info: http://owncloud.org/

It's a Dropbox alternative and self-hosted..
Mimi: Where'd you come from?
Doyle: My mom and the authorities are still trying to figure that out.
User avatar
Posts: 78
Joined: Mon Oct 22, 2012 1:25 pm
Location: The Netherlands
by ladidaaa » Wed Jan 30, 2013 5:39 pm
Hi,
Owncloud is nice, but is down when your server is down.
It only works with one-Server, and when this one is down no more file-syncing.
Also all the file-syncing has to be through the server, so the server has to copy the new file to every client.

BUT:
Luckily Bittorrentlabs seem to develop exactly the program i thought of, the call it Bittorrent SYNC:
http://labs.bittorrent.com/experiments/sync.html

This program syncs folders between an closed group of devices.
New devices can be added with a secret key, and syncing is done in Bittorrent style - so no main server that has to be always on. Also the connection is encrypted.
Review is here:
http://betanews.com/2013/01/28/test-driving-bittorrent-sync/
I just hope this program will one day be released and will work with a RPI ;-)
What do you think about that?
Posts: 11
Joined: Tue Jan 22, 2013 8:11 am
by JungleBoogie » Wed Jan 30, 2013 6:34 pm
Have you reviewed unison?
http://www.cis.upenn.edu/~bcpierce/unison/


rdiff is something else you may want to check out:
http://rdiff-backup.nongnu.org/
Happy Hacking!
Posts: 35
Joined: Thu Sep 06, 2012 12:06 am
Location: Simi Valley, California
by ladidaaa » Thu Jan 31, 2013 7:51 am
Thanks for the tip with Unison!
It seems to be a better alternative than Rsync - but the Unison-man describes if more than two devices are synced, there should be created a main "unison-hub".

This "main-hub" is exactly what i want to avoid for two reasons: slow-upload speeds on DSL-Connections and if this main-hub is down the whole "sync-network" is down.

I read on the Bittorrent - Sync Forum that one user already uses "Bittorrent Sync" with his RaspberryPi - so i suppose it will work when the final release whill be released.

I will try Bittorrent Sync once it is released and will post my findings here.
Posts: 11
Joined: Tue Jan 22, 2013 8:11 am
by andrewufrank » Sat Mar 08, 2014 11:41 am
there is btsync - which works well for exactly the purpose to build a network of peers (redundant) to synchronize files. i have 3 servers (RPi) and a number of other systems and all files are synced.
google for btsync - download either from http://www.bittorrent.com/sync/downloads or from https://github.com/tuxpoldo/btsync-deb

andrew
Posts: 1
Joined: Sat Mar 08, 2014 11:34 am