rsync to/from remote server - no password

[castletonroad]

Hi,

I use rsync to back-up root-access files from one server to another, using:

Code: Select all

sudo rsync -vazhPe ssh --rsync-path='sudo rsync' user@source:/path /path/

The 'user' on the remote and local machines is a member of the sudo group, so can execute rsync (and I have set 'user' in visudo to 'NOPASSWD' for rsync). I also have SSH passwordless login using SSH keygen enabled for 'user'.

However, the initial login to the remote server still prompts me for a password.

How can I securely feed a password to the command line above to avoid this.

Thanks very much.

[default_user8]

https://www.tecmint.com/ssh-passwordles ... asy-steps/

[castletonroad]

Thanks for the prompt response.

Apologies - I should have added that I already have SSH Passwordless Login Using SSH Keygen enabled and functioning. (I'll edit my post to reflect this.)

If this is all that is required, why am I still being prompted for a password when I execute the command above..?

[default_user8]

Hi,

I use rsync to back-up root-access files from one server to another, using:

Code: Select all

sudo rsync -vazhPe ssh --rsync-path='sudo rsync' user@source:/path /path/

The 'user' on the remote and local machines is a member of the sudo group, so can execute rsync (and I have set 'user' in visudo to 'NOPASSWD' for rsync). I also have SSH passwordless login using SSH keygen enabled for 'user'.

However, the initial login to the remote server still prompts me for a password.

How can I securely feed a password to the command line above to avoid this.

Thanks very much.

So can you ssh into the remote server at all without entering a password?

[castletonroad]

Yes 'user' can ssh between the two machines without passwords.

I am logged-in as 'user' on the local (destination) machine. I am assuming that it is also 'user' who is running rsync on the (remote) source, not 'root'?

[default_user8]

Yes 'user' can ssh between the two machines without passwords.

I am logged-in as 'user' on the local (destination) machine. I am assuming that it is also 'user' who is running rsync on the (remote) source, not 'root'?

I'm not exactly sure, but it appears to me that you are trying to run rsync on the remote machine as sudo, which to me is why you are having to provide a password. I found this older post that seems to pertain to your issue.
https://serverfault.com/questions/13654 ... both-sides

[castletonroad]

Hi, I do not think that’s the case.

The login is for accessing the remote machine (I get the host login prompt).

Once logged in, rsync runs without a sudo password request...

[bls]

rsync can be run on the remote system as a daemon, and the access can be constrained via /etc/rsyncd.conf to a reasonable, although not super-fine-grained granularity. For instance, my rsyncd.conf is:

Code: Select all

use chroot = true
log format = %h %o %f %l %b
pid file = /var/run/rsyncd.pid

[albums]
    path = /f/music/albums
    hosts allow = 192.168.16.0/24
    use chroot = false
    uid = bls
    gid = users
    read only = true
    dont compress = *

[bls]
    path = /home/bls
    hosts allow = 192.168.16.3, 192.168.16.11
    use chroot = false
    uid = bls
    gid = users
    read only = false
    dont compress = *

This rsyncd.conf creates two rsyncd modules, each with their own configuration. In this case, the remote user is 'bls', but I restrict access to these modules to specific hosts.

They're pretty similar, but you should be able to get the idea. 'man rsyncd.conf' will give you additional information. I start the rsyncd server using systemd socket, so the rsync server only runs when there is a connection. Here's /etc/systemd/system/rsyncd.socket

Code: Select all

[Unit]
Description=Rsync Server Socket
Conflicts=rsyncd.service

[Socket]
ListenStream=873
Accept=yes

[Install]
WantedBy=sockets.target 

I also mentioned this in my post about Raspberry Pi Lite-Er: https://www.raspberrypi.org/forums/view ... &p=1515887

[rpdom]

The reason you are getting the password response is because you are using sudo (at both ends).

I assume you have passwordless ssh set up for your normal (pi?) user.

When you run the rsync with sudo it runs as the root user, not pi, and it will try to connect to the remote system as root as well, so unless you have passwordless ssh set up for the root users it will ask for the password.

Having that set up for root can be a security risk: if someone gains access to one system as root, they will be able to ssh to the other system as root as well, but if your systems are secure they won't be able to get on at all. The advantage of running rsync as root is that all file permissions and ownerships are retained.

On servers I have worked on we did not have passwordless ssh for root, but we used the rsync daemon method described previously to give limited access instead. The rsync still ran as root, but was safer.
(In the end rsync wasn't the best method for what we were doing and I wrote a custom event triggered routine instead. rsync was better than the previous method that was used, but not really suited for trying to push updates on multiple servers to each other every few minutes. We often ended up with updates that were half complete.)

[castletonroad]

Thank you, @rpdom and @bls.

Your explanations and instructions make sense.

I'll have a go at setting rsync up to run as a daemon, and will see how I get on with that.

Thanks again.

[castletonroad]

So, I have rsync running as a daemon on 'raspberrypi4' (source), with /etc/rsyncd.conf as follows:

Code: Select all

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log

[nextcloud]
path = /mnt/ssd1TB_B/nextcloud
comment = RSYNC FILES - nextcloud
hosts allow = raspberrypi3
uid = pi
gid = users
read only = no
timeout = 300
don't compress = *

I then run the following on raspberrypi3 (destination) to 'pull' from raspberrypi4:

Code: Select all

sudo rsync -vazhPe pi@raspberyypi4::nextcloud/mnt/ssd1TB_B/nextcloud /mnt/ssd1TB_C/ --delete

The command runs, but only outputs the incremental file list, and tells me what the total size is (340GB) - no file sync'ing takes place.

I must be missing something still, but I've no idea what...

[scruss]

-e takes an argument (usually ssh) and you missed it off. The equivalent --rsh=COMMAND makes things clearer.

Can I suggest putting this in a script and using the long options (so instead of -vazhP, --verbose --archive --compress --human-readable --partial --progress) so you remember what each does? It helps prevent regressions from a working process to a not-working one.

Also, debug rsync commands by putting --dry-run as the very first option when you're testing, and only when the output looks good remove it? rsync's an amazing tool, but can really mess up a filesystem quickly: especially when using any of the --delete options as you are.
(It might have deleted all the unmatched files on the remote server instead of transferring anything, which appears to be what you asked it to do)

[castletonroad]

Thank you @scruss.

Yes, I forgot to remove the 'e' option (!). As per your suggestion, I have updated the command (not yet running from a script) and here is what I get:

Code: Select all

pi@raspberrypi3:/etc $ sudo rsync --verbose --archive --compress --human-readable --partial --progress pi@raspberrypi4::nextcloud /mnt/ssd1TB_C/nextcloud --delete --dry-run             receiving incremental file list
rsync: opendir "/data" (in nextcloud) failed: Permission denied (13)
IO error encountered -- skipping file deletion

sent 25 bytes  received 161 bytes  53.14 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1677) [generator=3.1.3]
pi@raspberrypi3:/etc $

I'm not sure that I have the rsync command written correctly, neither am I sure that my user permissions are set correctly...

I'd really welcome further help please (please!)...

[scruss]

I've never knowingly used rsyncd, but do you need the double colon '::'? ssh/scp typically only uses one: user@host:path

[bls]

I've never knowingly used rsyncd, but do you need the double colon '::'? ssh/scp typically only uses one: user@host:path

From the rsync man page: "Contacting an rsync daemon directly happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified "

Looks like the last rsync command that @castletonroad posted, with the long option names (great idea, BTW) is correct, but there appears to be a protection problem on the remote end. In the rsyncd.conf there is uid=pi,gid=users.

What is the uid/gid and protection of /mnt/ssd1TB_B/nextcloud?

[castletonroad]

@bls

The remote folder (and ideally destination folder) are www-data:www-data.

The start of this thread was because of 'permissions' issues (using sudo); the idea of using rsync as a daemon was to get around this.

The problem statement is: use a script on a destination machine to sync files/folders that are restricted access from a (remote) source machine, using rsync.

So any ideas where I go next?

Thanks for everyone's continued support resolving this.

[swampdog]

There's three potential hurdles.

(a) ensure the source "src" box can 'ssh' into the destination "dst" box without a password.
(b) unix file permissions.
(c) keeping it simple but with a mind to security.

Take user "foo" on "src" wants to copy root files to "dst"..

foo@src $ sudo id
^^^if that works then fine. Your first post indicated it does.

You essentially want..
foo@src $ sudo rsync --progress -auxv /wrk/ root@dst:/wrk/
^^^to keep dst:/wrk up to date with src:/wrk
Ignore my rsync options except to say I chose them because they are non-destructive to src:/wrk ("-c" for ((intensive)) checksum but man rsync).
^^^enable ssh root login on "dst" (/etc/ssh/sshd_config -> PermitRootLogin yes, restart sshd on dst. Possibly generate ssh key on root@dst (if not present already) then copy root@src:~/.ssh/id_rsa.pub (or whatever key) to dst and append to root@dst:~/.ssh/authorized_keys. Test..

root@src $ ssh root@dst
^^^should work without password.

That solves it for (a). Bear in mind it is a bad idea to allow passwordless root sshd login generally and the risk ramps up if either foo@src or root@dst is an internet facing machine. Having said that I do it myself inside my own network for the three hypervisors I run.

For (b) & (c) you might use a non root account for dst. It's a viable option when dst only needs to store an archive rather than access the files. You'd still be root on src but fling src:/wrk/ to dst:/wrk over ssh using (typically) tar such that bar@dst:/wrk ends up with a tarball, bar@dst:/wrk/src-wrk.tar.gz. Only requires passwordless login for bar@dst rather than root@dst.

There is also 'sshpass' which does what you ask (man sshpass). Very much frowned upon because it almost always gets abused by folk who don't care to think out (c) but under the right circumstances can be more secure (eg: adhoc jobs).

foo@src $ sshpass -p 'root@dst password' ssh root@dst
^^^noting that if 'ssh' asks any questions nothing will appear to happen (do 'ssh root@dst' manually - typically "Are you sure you want to continue connecting") and that with changing ip addresses the issue can randomly occur. You'll need extra options to 'ssh' to suppress that and I can't recall them. Last time I used 'sshpass' was to help a co-worker globally change a flag on a bunch of cisco devices .. many years ago. You'll note from the sshpass manpage even the author does not like his own tool. Perhaps he had to faff about with a load of cisco devices!

(a) is best compromise.

As you mention www-data the real solution is to match the uid/gid on the two boxes when the infrastructure was first designed. If both 'httpd' match then you can rsync directly using (say) apache@src -> apache@dst or even NFS (v4 - older is firewall nightmare).

[castletonroad]

@swampdog

Thanks very much for your detailed response.

I think my course of action is to issue the command on 'source', run the rsync daemon on 'destination', and change the permissions on the 'destination' folder.

Thanks very much for everyone's help, it's much appreciated.

[bls]

@bls

The remote folder (and ideally destination folder) are www-data:www-data.

The start of this thread was because of 'permissions' issues (using sudo); the idea of using rsync as a daemon was to get around this.

The problem statement is: use a script on a destination machine to sync files/folders that are restricted access from a (remote) source machine, using rsync.

So any ideas where I go next?

Thanks for everyone's continued support resolving this.

If the destination folder is owned by www-data:www-data, then you have two choices: 1) chown -R the remote folder to be pi:users, or change /etc/rsyncd.conf uid and gid lines to be uid=www-data and gid=www-data. They need to match, or the rsync daemon won't be able to write to the directory. Of course you change the protection on the folder tree to be 777, but that is usually undesirable.

[castletonroad]

@bls

That fixed things for me.

Code: Select all

pi@raspberrypi3:~ $ sudo rsync --verbose --archive --compress --human-readable --partial --progress pi@raspberrypi4::nextcloud /mnt/ssd1TB_C/nextcloud --delete

with /etc/rsyncd.conf:

Code: Select all

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
use chroot = true
log format = %h %o %f %l %b

[nextcloud]
    path = /mnt/ssdb1TB_B/nextcloud
    comment = RSYNC FILES - nextcloud
    hosts allow = raspberrypi3
    uid = www-data
    gid = www-data
    read only = no
    timeout = 300

...syncs the contents of my remote source folder (/mnt/ssdb1TB_B/nextcloud on raspberrypi4) into my local destination folder (/mnt/ssd1TB_C/nextcloud on raspberrypi3)

I tested this with the --dry-run option, but even that didn't stop me wiping out a critical folder on my destination!

All good now, and thanks everyone for your help.