Auto copy any device that is plugged in via USB

[nebulloyd]

Hello,

I have a relatively simple idea and wondering if anyone has done it, or if it can be done. I would like to set up a RP to automatically copy the data stored on any device when it is plugged into the USB port. I guess the data would be stored on an SD card in the easiest instance.

I would need advice from the ground up. i.e what type of RP, what bits and pieces I need, code, play by play steps and anything else that would be helpful. It would also be cool if any superstar programmer knows a way to make it only copy new data rather that repeating data that may be still on the device from the previous copy (not essential).

For some context this is a low energy cost solution for clearing up space on a GPS device for long trips in the forest recording monkey behaviour. RP would be powered with a portable battery. Thanks for your time.

[topguy]

It can be done.. I have seen threads here for doing the same thing but with images. Unfortunately I dont have a handy link to such a thread.

- It usually starts with "udev" rules that can be used to trigger a script when a device is inserted.
- Then its the work of the script to mount the drive, locate the files and copy them to a designated destination.
- a command like "rsync" can use "filenames", "modification date" or even "checksums" to avoid copying data already transferred.

[nebulloyd]

Thank you very much for taking the time that is a great help to give me new directions to investigate. I would also like to copy images. Do you think that an rsync could also specify file type?

[wildfire]

Do you think that an rsync could also specify file type?

That's one of rsyncs duties (to syncronise files), I suggest you

Code: Select all

man rsync

from a terminal.

[nebulloyd]

Hello,

I've been troubleshooting some connection issues but now I am back on this problem. I have access to my RPi3B via a VNC (using ethernet to macbook pro).

Would you be able to break down the steps in doing setting up the automated file sync a little more? I am a first timer on a steep learning curve.

I used

Code: Select all

lsusb

to get the info for my GPS which reads

Code: Select all

Bus 001 Device 004: ID 091e:2459 Garmin International GPSmap 62/78 series

I now need to know how to:

1. make the udev rule
2. how to run the rsync script when the device is detected (to store certain files from GPS to preferrably a USB drive or otherwise the kernel mSD card)
3. Auto shutdown device with some kind of signal that it is complete

[karrika]

The first thing is to create the udev rule like /etc/udev/rules/99-usbcheck.rules

It could contain something like:

Code: Select all

KERNEL!="sd[b-z]1", GOTO="usbcheck_end"
IMPORT{program}="/sbin/blkid -o udev -p %N"
ACTION=="add", ENV{mount_options}="relatime"
ACTION=="add", ENV{ID_FS_TYPE}=="vfat|ntfs", ENV{mount_options}="$env{mount_options},utf8,gid=100,umask=002"
ACTION=="add", \
 RUN+="/usr/local/bin/blink", \
 RUN+="/bin/mkdir -p /tmp/usb", \
 RUN+="/bin/chmod a+rwx /tmp/usb", \
 RUN+="/bin/mount -o $env{mount_options} /dev/%k /tmp/usb", \
 RUN+="/usr/local/bin/processusb" \
 RUN+="/usr/local/bin/light"
ACTION=="remove", RUN+="/bin/umount -l /tmp/usb", \
 RUN+="/bin/chmod a-w /tmp/usb", \
 RUN+="/usr/local/bin/dark"
LABEL="usbcheck_end"

Here you have three routines for blinking a LED:
/usr/local/bin/blink
/usr/local/bin/light
/usr/local/bin/dark

You have to write these depending on where your LED is.

To do the copy this script will execute /usr/local/bin/processusb. You have to write that as well.

The processusb could contain a rsync command like below.

Code: Select all

/usr/bin/rsync -rptuv --modify-window=1 --partial --force /tmp/usb/ /opt/USBcontent

The strange --modify-window=1 is needed because USB sticks using VFAT have a time resolution of 2s. This line tells Linux to ignore small errors in the timestamp +- 1s.

In the "remove" action you can also put the shutdown command if you want.

[nebulloyd]

A great reply, thanks for that!

ok so i made the file using

Code: Select all

sudo nano /etc/udev/rules.d/71-garmingps.rules will that be ok?

I then subbed out 'usbcheck' for 'garmingps' in your code.

assuming that is all correct...

where do I put this command /usr/bin/rsync -rptuv --modify-window=1 --partial --force /tmp/usb/ /opt/USBcontent

[karrika]

The rsync goes into a file called /usr/local/bin/processusb

You also need to make this file executable and add a proper header in front of it like this:

Code: Select all

#!/bin/bash
/bin/mount -o remount,async,noatime,norelatime /tmp/usb
/usr/bin/rsync -rptuv --modify-window=1 --partial --force /tmp/usb/ /opt/USBcontent

Code: Select all

sudo chmod +x /usr/local/bin/processusb

Working with udev stuff is not really for beginners. Try to experiment a little and understand the lines what they do. This is kind of advanced stuff.

[nebulloyd]

so..

Screen Shot 2016-11-13 at 9.36.40 pm.png (32.82 KiB) Viewed 4268 times

Screen Shot 2016-11-13 at 9.37.27 pm.png (27.07 KiB) Viewed 4268 times

like this?

Yes, I am definitely in over my head but happy to stick with it so long as people have patience thanks again for your time I really appreciate it

[karrika]

Wow. Perhaps it would be good to start from very basics like how to enter a command on the console.

You can enter a command and separate it from its argument with a space.

Code: Select all

cd/media

Does not separate the command from the argument.

Code: Select all

cd /media

does separate the command from the argument.

Try to find the other similar errors in your previous post. The command "chmod" is used to change attribute bits on a file.

Typical attributes are
r = read
w = write
x = executable ( like a .bat file in Windows )

The command sudo is to gain super user power for executing the next command.

There is a lot more about access control. The obvious thing is that chmod was not part of the content of processusb. It was a command to change processusb to an executable script file.

[topguy]

Based on previous experiences with udev, the "processusb" script should probably spawn off the rsync process in the background. The process will be killed by linux/systemd if it runs for too long (more than a few seconds). This was a problem for those who copied lot of imagefiles.

I would probably also have moved the mount commands and LED control also to a new script together with the rsync.

[karrika]

My example is old before systemd and jessie.

You can run stuff in the background by surrounding the code with ( ) and adding a & after it.

Code: Select all

(
  command1
  command2
  ...
) &

Then the udev rule can exit immediately and the code keeps running in the background. This is a bad solution as you do not have control on what goes on. A better solution might be to create a service of the copying process and command it from the systemctl. But that is quite complicated.

[DougieLawson]

A better solution might be to create a service of the copying process and command it from the systemctl. But that is quite complicated.

That's trivial with systemd.

sudo nano /etc/systemd/system/usbcopy.service

Code: Select all

[Unit]
Description=Run usbcopy cmd

[Service]
ExecStart=/usr/local/bin/usbcopy.sh
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=usbcopy
Type=oneshot

Enable that with systemctl enable usbcopy.service
Start that with systemctl start usbcopy

[nebulloyd]

Thank you so much for your time in responding. I hope to meet up with someone from IT at my uni on Monday. I will post as much as I can remember about how she does it, I hope it can be done!