Failing to copy files to USB

[ppparadox]

Hi, i'm having issues with my raspberry pi model B (running a fully up-to-date raspbian) and my 8GB USB storage device: i am developing a small embedded system which has to check for the presence of certain files inside the USB device and copy them (if they're not there). My problem is that only few (3-5) out of ~20 files in dir1 are actually being copied to USB. What can i do?

It all happens via bash script (this are just the last few lines):

Code: Select all

if [[ ! -d "$STORAGE"/dir1 ]]; then
    cp -rf "$SOURCE"/dir1 "$STORAGE"
fi
sleep 10
sync
sleep 20
sync
shutdown -h now

#lsusb -v output:

Code: Select all

Bus 001 Device 005: ID 0718:0619 Imation Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0718 Imation Corp.
  idProduct          0x0619 
  bcdDevice            1.00
  iManufacturer           1 Imation 
  iProduct                2 Nano Pro        
  iSerial                 3 [redacted]
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval

[cpc464]

Well what error do you see when running the script ? Also, is there any particular reason for using the -f flag on cp.

[ppparadox]

No errors, the -f flag is just to overwrite without prompting.

[cpc464]

Couple of things. The "-f" flag is not for overwriting I think. cp should overwrite by default. However, with your code, the issue of overwriting should never arise, since you check that $STORAGE/dir1 does not exist before newly creating $STORAGE"/dir1 with the cp command.

Try using the "-v" flag with cp to see what is happening. And maybe put the "-x" flag at the start of your script, to see commands a they are run.

[ppparadox]

Been there, done that. No problem whatsoever. Also as i said the problem is that some files get copied, not all of them, which smells like some ugly form of caching is in the way.

[ilguargua]

I would try to umount the usb device before shutdown, maybe can help, even if the system should properly umount all devices before shutting down...


Cheers, Ale

[cpc464]

I am not sure what you mean by "been there, done that". Disk caching is unlikely to be the culprit. If cp is failing to copy files which you think it should be copying, cp -v, #!/bin/bash -x will show why. dmesg or tail /var/log/messages or syslog will show any hardware errors.

[ppparadox]

Ok i did some more testing and now i think i've narrowed down the problem: if i run the script without the final "shutdown -h now" everything is OK.
Still it is very worrying that i'm losing data even after TWO calls to sync and a proper shutdown. How come?

[DougieLawson]

What's the filesystem on the USB stick? Have you tried explicitly umounting it (to ensure it's buffers are flushed)? Have you considered reformatting it to ext4?

[ppparadox]

File system is FAT32, unmounting before shutdown is the only thing i haven't tried yet. Reformatting is out of the question because of the nature of what i'm trying to accomplish (windows compatibility yay). But is there that big a difference from calling sync?

[ilguargua]

Please try, and le us know the answer!


Cheers, Ale.

[DougieLawson]

File system is FAT32, unmounting before shutdown is the only thing i haven't tried yet. Reformatting is out of the question because of the nature of what i'm trying to accomplish (windows compatibility yay). But is there that big a difference from calling sync?

https://bugs.debian.org/cgi-bin/bugrepo ... bug=579575
suggests using mount -t vfat -o flush,other_options_here ... to force a flush when the copy is complete.

[ppparadox]

I managed to solve the issue. The situation was the following, a bit more complicated than i initially said:
/etc/rc.local contained calls to several scripts.
The first one was meant for the first boot only and after a few things had to shutdown but the very next script was the one containing the problematic file checking. So after the first script had successfully exited (after calling shutdown) the next one had just started copying those files but didn't have enough time to finish before being terminated by the pending shutdown.
Upon rebooting the second script would have been called again (not the first, that was for the first boot only) but the folder was there as far as if [[ ! -d "$folder" ]] was concerned... just not all of the files inside.
Maybe it would have been better to use rsync and check for each and every file.
Still worrying though. I thought file copying was more reliable than this.
Thanks to all of you for your kind answers!