dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

rsync - works from terminal but not as cron???

Thu Jul 18, 2019 9:46 pm

I have set up an rsync to back up my weather data from my pi (3B) to my PC. Here is the command:
sync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public
I can get it to run no problem from the PC, but when I set it up as a cron job in the GUI schedular on my Pi it never completes!.....the log says
Jul 19 08:00:01 WeatherPi CRON[8466]: (pi) CMD (rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public # JOB_ID_2)
What am I missing?

LTolledo
Posts: 1768
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: rsync - works from terminal but not as cron???

Thu Jul 18, 2019 9:51 pm

can you post the contents of your cron job?
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Thu Jul 18, 2019 10:03 pm

LTolledo wrote:
Thu Jul 18, 2019 9:51 pm
can you post the contents of your cron job?
OK, where do I find that? Does the log record not show the command? If you mean the data that is being transferred, it is a the Dupliciy (Backup) output of my weather data.

netview
Posts: 4
Joined: Tue Jul 02, 2019 10:15 am

Re: rsync - works from terminal but not as cron???

Thu Jul 18, 2019 10:41 pm

Try this: 'sync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public &&'
you have to add '&&' at the end of the command!

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Thu Jul 18, 2019 11:14 pm

netview wrote:
Thu Jul 18, 2019 10:41 pm
Try this: 'sync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public &&'
you have to add '&&' at the end of the command!
Is this from the command line or from the cron job?

I used the cron and added the && to the end and then ran it....this is what comes up on the messages log.

Jul 19 11:14:58 WeatherPi vncserver-x11[496,root]: Connections: connected: 10.1.1.155::37322 (TCP)
Jul 19 11:14:58 WeatherPi vncserver-x11[496,root]: Connections: authenticated: 10.1.1.155::37322 (TCP), as (anonymous) (d permissions)

bls
Posts: 211
Joined: Mon Oct 22, 2018 11:25 pm

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 2:54 am

You could force some rsync logging to see if that helps in determining where it's hanging up. Change your rsync command to:

Code: Select all

rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public 2>&1 >> /home/pi/rsync-backup.log
Of course, change /home/pi/rsync-backup.log as you'd prefer.

If you try this and need further help, please post the last 20 or so lines of the log file here.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 5:10 am

Code: Select all

rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public 2>&1 >> /home/pi/rsync-backup.log
Running this as a Cron (i.e forcing it to run under Scheduler) never even resulted in the formation of rsyc-backup.log?!?!?!

User avatar
Paeryn
Posts: 2634
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 5:46 am

dunbrokin wrote:
Fri Jul 19, 2019 5:10 am

Code: Select all

rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public 2>&1 >> /home/pi/rsync-backup.log
Running this as a Cron (i.e forcing it to run under Scheduler) never even resulted in the formation of rsyc-backup.log?!?!?!
The order of redirects that bls said is the wrong way around, stderr was set to the original stdout (which cron throws away if it can't email it to you) and then sets stdout to append to the file. So as given only stdout would go to the file, any errors written to stderr would still get silently lost, try this way around, this sets stdout to append to the file first and then sets stderr to the same file as stdout..

Code: Select all

rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public >> /home/pi/rsync-backup.log 2>&1
She who travels light — forgot something.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 6:01 am

.

Code: Select all

rsync -a -e ssh /home/pi/Public/ [email protected]:/home/pj/Public >> /home/pi/rsync-backup.log 2>&1

Sorry to report.....this fared no better?!?!?!

tpyo kingg
Posts: 586
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 6:43 am

dunbrokin wrote:
Thu Jul 18, 2019 9:46 pm
What am I missing?
How are you telling it which SSH key to use? I'm not seeing an -i option for the SSH client.

Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/some_key_ed25519' /home/pi/Public/ [email protected]:/home/pj/Public
If you want to use an agent as well, that will mean a little more set up.

And / or, once rsync is working via cron, you can lock down the key over in authorized_keys on the remote machine by prepending command="..." to the public key. You can see the command to put there by adding -v to the SSH client's options.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 9:55 am

How are you telling it which SSH key to use? I'm not seeing an -i option for the SSH client.

Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/some_key_ed25519' /home/pi/Public/ [email protected]:/home/pj/Public
But it works from the command line in the same format......but not from the Cron......why should the -i option make a difference to that?!?!

tpyo kingg
Posts: 586
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 10:35 am

dunbrokin wrote:
Fri Jul 19, 2019 9:55 am
But it works from the command line in the same format......but not from the Cron......why should the -i option make a difference to that?!?!
The environment variables in cron are different than the ones in an interactive shell session.

LTolledo
Posts: 1768
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 10:59 am

dunbrokin wrote:
Thu Jul 18, 2019 10:03 pm
LTolledo wrote:
Thu Jul 18, 2019 9:51 pm
can you post the contents of your cron job?
OK, where do I find that? Does the log record not show the command? If you mean the data that is being transferred, it is a the Dupliciy (Backup) output of my weather data.
as root run the following command:

Code: Select all

~# crontab -l
and post the lines that does not start with #

below is a sample of an entry in my crontab,
this entry works, syncing 2 network drives from 2 different SBCs
the netsynch.sh file contain the rsync command syntax for the syncing of 2 network drives

Code: Select all

0 21 * * * /home/pi/netsync.sh &
I'd like to see if something similar "exist" in your crontab.
if none, ... well ....you get the idea...
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 11:03 am


The environment variables in cron are different than the ones in an interactive shell session.
OK, thanks for that, was not aware of it...... So where/hoe do I find the variable "some_key_ed25519"?:

Code: Select all

/.ssh/some_key_ed25519

bls
Posts: 211
Joined: Mon Oct 22, 2018 11:25 pm

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 5:35 pm

Paeryn wrote:
Fri Jul 19, 2019 5:46 am

The order of redirects that bls said is the wrong way around.
Good catch on my reversed redirections. Oops :(

As penance for my oops, I'll offer another method of using rsync that might be simpler for the OP, if he has control of the rsync remote system: Using an on-demand rsync service.

Here's how:

1) Create /lib/systemd/system/rsyncd.socket: sudo nano /lib/systemd/system/rsyncd.socket
Paste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

[Unit]
Description=Rsync Server Socket
Conflicts=rsync.service

[Socket]
ListenStream=873
Accept=yes

[Install]
WantedBy=sockets.target 
2) Create /lib/systemd/system/[email protected]: sudo nano /lib/systemd/system/[email protected]
Paste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

[Unit]
Description=Start the rsync server daemon
After=network.target
ConditionPathExists=/etc/rsyncd.conf

[Service]
ExecStart=/usr/bin/rsync --daemon --no-detach
StandardInput=socket
3) Create /etc/rsyncd.conf with the following contents, adjusted as appropriate for your configuration: sudo nano /etc/rsyncd.conf
P
aste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

use chroot = true
log format = %h %o %f %l %b

[weatherdata]
    path = /home/pj/Public
    hosts allow = 10.1.1.10, 127.0.0.1    << Adjust this IP address for the rsync source system. Can also be a range, eg. 10.1.1.0/24
    use chroot = false
    uid = pj                              << Adjust this as needed for the username on the receiving system
    gid = users                           << Ditto
    read only = false
    dont compress = *
4) bash$ sudo systemctl daemon-reload
5) bash$ sudo systemctl enable rsyncd.socket
6) bash$ sudo systemctl start rsyncd.socket

Then, on the remote system, you can do: bash$ rsync -a /home/pi/public 10.1.1.1::weatherdata

This configuration will cause systemd to start the rsync daemon on demand. Each time an rsync connection is made, systemd starts a unique rsync daemon using the contents of [email protected]

For a home configuration this should be satisfactory from a security perspective. If security is more of a concern, you can tighten up the configuration in /etc/rsyncd.conf, see 'man rsyncd.conf' for details

tpyo kingg
Posts: 586
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: rsync - works from terminal but not as cron???

Fri Jul 19, 2019 6:16 pm

dunbrokin wrote:
Fri Jul 19, 2019 11:03 am
So where/hoe do I find the variable "some_key_ed25519"?:
It's a stand-in for the path and name of the actual key you are using to log in. When you ran ssh-keygen to make the keys, which parameters did you use? The usual place to keep SSH keys for the user pi is in the directory /home/pi/.ssh/

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Sat Jul 20, 2019 4:06 am

It's a stand-in for the path and name of the actual key you are using to log in. When you ran ssh-keygen to make the keys, which parameters did you use? The usual place to keep SSH keys for the user pi is in the directory /home/pi/.ssh/
Ok, well, in my .ssh folder there are 3 files: 'id_rsa', 'id_rsa.pub' and 'known_hosts'. All of them contain keys which are 50k characters long (i exaggerate for effect!) none has anything which resembles 'some_key_ed_abc123'

I set it up at the start to be without any password....so just CR when asked for password.

User avatar
rpdom
Posts: 14987
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: rsync - works from terminal but not as cron???

Sat Jul 20, 2019 4:31 am

The id_rsa is the file you would use for that option, but that should be picked up as the default anyway without having to specify it.

I have no idea about this "GUI schedular" that you used. I just know simple crontab files. Is the command running from the standard "pi" crontab?

tpyo kingg
Posts: 586
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: rsync - works from terminal but not as cron???

Sat Jul 20, 2019 6:04 am

Changed the order, because of importance.
dunbrokin wrote:
Sat Jul 20, 2019 4:06 am
I set it up at the start to be without any password....so just CR when asked for password.
Wait. Having to press the Enter key means that authentication is not going automatically. Usually when using SSH keys the way you are trying to it should ask for neither or if it does ask for a passphrase, just the first time per session. When connecting manually does it ask for either a pass word or a pass phrase?
dunbrokin wrote:
Sat Jul 20, 2019 4:06 am
Ok, well, in my .ssh folder there are 3 files: 'id_rsa', 'id_rsa.pub' and 'known_hosts'. All of them contain keys which are 50k characters long (i exaggerate for effect!) none has anything which resembles 'some_key_ed_abc123'
The known_hosts is (mostly) unrelated. It is a register of the public keys from the remote hosts you have already connected to. That would include your Raspberry Pi. See the "FILES" section in the manual page for sshd about that one.

The 'id_rsa', 'id_rsa.pub' are the relevant ones. They are the default names for SSH key pair in the default SSH key format (RSA). You'd specify it a script or cron job like this:

Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/id_rsa' /home/pi/Public/ [email protected]:/home/pj/Public/
Please see if pointing to the key in the cron job works, if it also works outside in the normal, interactive shell.
If it does not work, then try increasing the verbosity of the client and catching that chatter in a separate file. Here is anothe way to do that:

Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/id_rsa -v -E /home/pi/rsync-backup.log' /home/pi/Public/ [email protected]:/home/pj/Public/
By the way, ssh-keygen has the -f and -C options to make custom file names for the keys and add comments to them. Sadly most guides and tutorials regarding SSH keys are badly lacking on the topic of key managment.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Sun Jul 21, 2019 12:07 am

[
Wait. Having to press the Enter key means that authentication is not going automatically. Usually when using SSH keys the way you are trying to it should ask for neither or if it does ask for a passphrase, just the first time per session. When connecting manually does it ask for either a pass
Sorry did not make myself clear here. On initial set up of the key, I chose enter so that I would not need a password. When I run manually it does not ask for anything...it just successfully executed the command.

Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/id_rsa' /home/pi/Public/ [email protected]:/home/pj/Public/
This works perfecly at the command line, but again does not succeed under Cron.


Code: Select all

rsync -a -e 'ssh -i /home/pi/.ssh/id_rsa -v -E /home/pi/rsync-backup.log' /home/pi/Public/ [email protected]:/home/pj/Public/
This produced the log file....but the log file contains nothing.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Sun Jul 21, 2019 12:12 am

rpdom wrote:
Sat Jul 20, 2019 4:31 am

I have no idea about this "GUI schedular" that you used. I just know simple crontab files. Is the command running from the standard "pi" crontab?
This is the scheduler:

http://gnome-schedule.sourceforge.net/

I have not had time to run it under simple cron.....but it does not work in the scheduler (which, one assumes, is the same thing as cron!).

Indeed if I enter crontab -e i get the same code that I entered into the Gui.

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Mon Jul 22, 2019 9:27 am

bls wrote:
Fri Jul 19, 2019 5:35 pm
Paeryn wrote:
Fri Jul 19, 2019 5:46 am

The order of redirects that bls said is the wrong way around.
Good catch on my reversed redirections. Oops :(

As penance for my oops, I'll offer another method of using rsync that might be simpler for the OP, if he has control of the rsync remote system: Using an on-demand rsync service.

Here's how:

1) Create /lib/systemd/system/rsyncd.socket: sudo nano /lib/systemd/system/rsyncd.socket
Paste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

[Unit]
Description=Rsync Server Socket
Conflicts=rsync.service

[Socket]
ListenStream=873
Accept=yes

[Install]
WantedBy=sockets.target 
2) Create /lib/systemd/system/[email protected]: sudo nano /lib/systemd/system/[email protected]
Paste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

[Unit]
Description=Start the rsync server daemon
After=network.target
ConditionPathExists=/etc/rsyncd.conf

[Service]
ExecStart=/usr/bin/rsync --daemon --no-detach
StandardInput=socket
3) Create /etc/rsyncd.conf with the following contents, adjusted as appropriate for your configuration: sudo nano /etc/rsyncd.conf
P
aste the following text in, and then exit the editor with CTRL/X, Y, ENTER

Code: Select all

use chroot = true
log format = %h %o %f %l %b

[weatherdata]
    path = /home/pj/Public
    hosts allow = 10.1.1.10, 127.0.0.1    << Adjust this IP address for the rsync source system. Can also be a range, eg. 10.1.1.0/24
    use chroot = false
    uid = pj                              << Adjust this as needed for the username on the receiving system
    gid = users                           << Ditto
    read only = false
    dont compress = *
4) bash$ sudo systemctl daemon-reload
5) bash$ sudo systemctl enable rsyncd.socket
6) bash$ sudo systemctl start rsyncd.socket

Then, on the remote system, you can do: bash$ rsync -a /home/pi/public 10.1.1.1::weatherdata

This configuration will cause systemd to start the rsync daemon on demand. Each time an rsync connection is made, systemd starts a unique rsync daemon using the contents of [email protected]

For a home configuration this should be satisfactory from a security perspective. If security is more of a concern, you can tighten up the configuration in /etc/rsyncd.conf, see 'man rsyncd.conf' for details
Thanks for that, just to clarify...I have the data on the pi and need to transfer to the PC - which (just to add to the confusion is called pj). Thank you for your offering above. Just before I implement it, could you please show which device each of the numbered sequences should be perfomed on. Thank you!

bls
Posts: 211
Joined: Mon Oct 22, 2018 11:25 pm

Re: rsync - works from terminal but not as cron???

Mon Jul 22, 2019 1:32 pm

dunbrokin wrote:
Mon Jul 22, 2019 9:27 am

Thanks for that, just to clarify...I have the data on the pi and need to transfer to the PC - which (just to add to the confusion is called pj). Thank you for your offering above. Just before I implement it, could you please show which device each of the numbered sequences should be perfomed on. Thank you!
Ah, I missed the point that one of the systems was not a Pi. what operating system is running on "pj"?

dunbrokin
Posts: 29
Joined: Sat Jan 05, 2019 3:16 am

Re: rsync - works from terminal but not as cron???

Mon Jul 22, 2019 8:17 pm


Ah, I missed the point that one of the systems was not a Pi. what operating system is running on "pj"?
Pc running on Ubuntu 19.04. Thanks.

bls
Posts: 211
Joined: Mon Oct 22, 2018 11:25 pm

Re: rsync - works from terminal but not as cron???

Mon Jul 22, 2019 9:03 pm

dunbrokin wrote:
Mon Jul 22, 2019 8:17 pm

Ah, I missed the point that one of the systems was not a Pi. what operating system is running on "pj"?
Pc running on Ubuntu 19.04. Thanks.
Ubunto is good! The directions should work for you. The entire set of steps is done on the Ubuntu system up through and including enabling and starting rsyncd.socket. Then, to test from the Pi, type rsync 10.1.1.155::weatherdata . You should get something resembling a directory listing if it's working. If it doesn't work, check the journalctl log on the Ubuntu system for additional details.

Once it's all working, based on your original posting, you should use rsync -a /home/pi/Public/ 10.1.1.155::/home/pj/Public.

Return to “Troubleshooting”