Raspberry Pi as USB thumb Drive is not been detected by the host peripheral when disconnected without ejecting

[Linux_01]

Hello All,

I have configured Raspberyy Pi Zero as USB Thumb Drive.
Upon connected to the Macbook, it gets detected and I am am able to transfer files from MAC to the drive. And the same has been accessed at the Pi end as well.

The Problem starts when I disconnect the USB Thumb Drive from Macbook without ejecting it and try to connect again.
If I reconnect the drive to MacBook, it does not detect the USB drive.

At Raspberry Pi Zero end, in the logs its show:
Mar 13 15:10:07 Jha-RPi-USBDrive-01 kernel: [ 1437.308879] dwc2 20980000.usb: new device is high-speed
Mar 13 15:10:07 Jha-RPi-USBDrive-01 kernel: [ 1437.320850] dwc2 20980000.usb: new address 10

Please provide me some pointers to troubleshoot this issue.

Thank you.

[Linux_01]

Any pointer will help me to move further.

[Imperf3kt]

Always, without fail, eject before removal.
The feature exists to prevent file and card corruption, which could be causing what you're seeing.

Do that, and your issue won't ever present itself.

[Linux_01]

I am looking for a solution wherein if I dont eject the USB then after If I reconnect the USB, it should detect it.

Do we have any mechanism by which we can achieve the same?

[thagrol]

I am looking for a solution wherein if I dont eject the USB then after If I reconnect the USB, it should detect it.

Do we have any mechanism by which we can achieve the same?

Almost certainly not. Access from the USB host is low level access via a subset of SCSI commands (that's how USB mass storage works). The USB host will be doing it's own read and write caching (linux and windows do, I assume Macs do too but don't have one) so without ejecting the drive everything in those caches will be lost.

Depending on what you're using for a backing store for the mass storage gadget there might be some data in the linux side caches too but there's no guarantee that such data will be complete.

Another thing that might be relevant is how you're powering the zero. Only from the Mac or via a seperate PSU. If with a seperate PSU (or via two cables to the Mac - one for power one for data) it's possbile that the Pi end of the USB gadget gets into a confused state though I've not seen it do so on linux or windows hosts.

If you use one cable from the Mac for power and data, the pi will reboot with clean caches etc when connected but be aware that by pulling the plug you risk SD card corruption as the OS won't be cleanly shutdown even if the drive is ejected from the Mac.

You could try setting your Mac to not use read or write caching on the USB gadget. If using a file on the zero as backing store you'll also need to mount the filesystem containing that file with the sync option so there's no caching on the linux side.

[Linux_01]

@thagrol, thank you for the quick response.

Now there two concerns:

A) I have found that when I load the drive with read only, I am able to disconnect and reconnect the USB drive at Host side:

Code: Select all

modprobe g_mass_storage file=/piusb.bin stall=0 ro=1 removable=1

But if I make it read and write then I am not able to reconnect the USB Drive after disconnecting it without ejecting at the host side.

So wanted to know is there any workaround to reconnect the drive successfully or am I missing something?

B) After copying data into the USB Drive, at Pi Zero side I am do see the data unless and until I unmount and remount the drive.

Is it possible to view the contents on the go? Again am I missing something.

[thagrol]

@thagrol, thank you for the quick response.

Now there two concerns:

A) I have found that when I load the drive with read only, I am able to disconnect and reconnect the USB drive at Host side:

Code: Select all

modprobe g_mass_storage file=/piusb.bin stall=0 ro=1 removable=1

But if I make it read and write then I am not able to reconnect the USB Drive after disconnecting it without ejecting at the host side.

So wanted to know is there any workaround to reconnect the drive successfully or am I missing something?

Not that I'm aware of. Might be Mac OS thing.

B) After copying data into the USB Drive, at Pi Zero side I am do see the data unless and until I unmount and remount the drive.

Is it possible to view the contents on the go? Again am I missing something.

No, and yes.

This is going to get technical...

host: the USB host
gadget: the zero running as a USB gadget

The main problem is that both the host and gadget have completely seperate views of the backing store.

They do not communicate when changes are made.

All the gadget knows is when the host writes a block to the backing store but this is only known to the g_mass_storage module not to the OS on the gadget.

It gets worse.

Despite what the filesystem and OS on each end show actuall access to the storage is at the block level.

Both the host and the gadget use read and write caching.

Both the host and the gadget expect to have exclusive access to the shared storage.

While the g_mass_storage module knows when a block has been written by the host, it does not know whether that was the last block nor does it know whether the host has further data in its write cache. As mentioned above, it does not pass this knowledge on to the gadget's OS.

So, the only way to see changes on the gadget side is to unmount and remount but there is no guarantee that you will see all the changes as data may still be in the host's write cache.

And don't even think about writing to the shared storage from both sides at the same time. That's a sure way to corrupt the shared storage and confuse the side not writing.

You might see different behaviour if you disble read and write caching on both the host and the gadget when mounting the shared storage but I've not tested that.

The safest way to get data from one side to the other using a mass storage gadget is to only mount it on the writing side, write all the data you're going to, unmount/eject it, then mount it on the other side.

If you must have the shared storage mount by hte host and the gadget at the same time the only safe way is for both to do so read only.

[Linux_01]

Let me explain my Requirement.

I am not writing simultaneously to the USB Drive.

I am developing this Raspberry Pi zero as USB thumb drive so that I can use it as pendrive with Datalogger machine so that it can continuously throw the data to the Pi.

Datalogger has only one option to send data that is through USB port.

So I want that even if an user removes pi from datalogger upon reconnecting it should start sending data to the Pi.

[thagrol]

I have no experience of data loggers and obviously don't have one.

Behaviour on your Mac may not be the same as behaviour on your datalogger. You need to test on that.

I'm afraid there isn't much more I can offer in the way of help, if anything.

[PiGraham]

Maybe it would help if you set Raspbian as read-only on the Pi0. If read access can be shared safely and the host needs write access then removing write access from Raspbian might help.

There is an option for than in Buster.
viewtopic.php?f=63&t=253104&p=1549229#p1549117

OTOH that might prevent the host write as well. Try it and see?

[thagrol]

Maybe it would help if you set Raspbian as read-only on the Pi0. If read access can be shared safely and the host needs write access then removing write access from Raspbian might help.

There is an option for than in Buster.
viewtopic.php?f=63&t=253104&p=1549229#p1549117

That's likely to do more harm than good. It looks to me that the OP is using a file for the backing store and that file is located on the root partition. Making that partition read only will also prevent the USB host from writing to the shared storage.

Making raspbian read only may be sensible if end users are likely to just pull the zero with doing a proper shutdown first, but the backing store needs to be either a writeable partition or on a writeable partition or teh USB host won't be able to write to it regardless of settings in g_mass_storage and on the USB host.

[PiGraham]

The backing store surely has to be on a writable partition that is not being accessed by the OS.

Partition the SD card then make Raspbian R-O?

[thagrol]

The backing store surely has to be on a writable partition that is not being accessed by the OS.

Partition the SD card then make Raspbian R-O?

Not necessarily.

The backing store doesn't have to be a partition. It can be a file, in which case the partition containing that file must be writeable by the zero's OS or the host won't be able to write to it.

So, yes. TO make the OS read only you'd need a third partition for the backing store whether you're using a file or directly using that partition.

The USB host sees the same thing whether using a file or a partition though I've some issues when using a partition that don't occur when using a file.

Where things can get confusing with using a partition is than depending on the OS on the host and where/how you format the backing store it can end up with it's own boot sector and partition table. Or the otherway around: a formatted block device that appears to the host but that doesn't have a partition table at all.

If you format the partition on the gadget under raspbian, it'll be a formatted volume but won't have a partition table (or mbr/gpt record).

If you leave it unformatted on raspbain then format it on the USB host, the host will treat it as a bare drive and create MBR/GPT stuff and a partition table.

It's my experience that this will confuse some OS if you later try to access the card via a card reader. Or try to mount that shared partition.

[PiGraham]

OK. So no options for the OP?
Unsafe removal not only risks data loss but also OS corruption and the reported failure to remount on re-insertion.

You sound familiar with this. Have you tried sudden removal and re-insertion of a Pi0 as mass storage gadget?

I'm assuming this is not an issue with standard USB memory so it's something do with the Pi0.

[PiGraham]

If removal trashes the backing file might that prevent remounting? If that's the case is there a check to confirm it? If theres a problem with the backing file might creating a new file allow remounting?

[thagrol]

OK. So no options for the OP?
Unsafe removal not only risks data loss but also OS corruption and the reported failure to remount on re-insertion.

OS corruption is only a risk with unsafe removeal if the zero is not seperately powered. It's also a risk if the zero is only powered from the host and the host is switched off without first shutting down the Pi.

However, powering a zero from a USB host and a seperate PSU has it's own problems as all the 5v pins are directly connected to each other on the zero(w).

You sound familiar with this. Have you tried sudden removal and re-insertion of a Pi0 as mass storage gadget?

I've done a few projects with various USB gadgets, yes. Not done any sudden removal testing recently or to any major extent. Most of the issues are not specific to the mass storage gadget, they're related to how modern file systems and disc access work.

I'm assuming this is not an issue with standard USB memory so it's something do with the Pi0.

Not really. Sudden removal is a problem for any storage device where the host (and sometimes the device itself) uses write caching. It's not specific to USB or the Pi zero(w).

The potential for OS corruption is however specific to Pi and similar devices. Your average USB stick/memory card/HDD/SDD doesn't have a full OS running on it behind the scenes.

[thagrol]

If removal trashes the backing file might that prevent remounting? If that's the case is there a check to confirm it? If theres a problem with the backing file might creating a new file allow remounting?

We don't know for certain that it is trashing the backing store. It might be something related to Mac OS.

While we know tha Mac isn't mounting the device, we don't know if the g_mass_storage gadget is still running and whether the Mac is detecting the device at any level.

Over to the OP. Can you check that please.

As for checking the file system, if the Mac detects the device try scanning it there otherwise try unmounting the backing storage on the zero, creating a loop device for it (as it's a file) then running fsck on it.

[Linux_01]

When I connect the Pi as USB thumb drive for the 1st time to the MAC book, it detects and able to transfer the data.

If I disconnect the Pi from MAC book without ejecting it an dthen reconnect MAC doesn't detect where as at Pi end logs shows it detects the reinsertion but cannot map:

Code: Select all

Mar 17 16:41:35 Jha-RPi-USBDrive-01 kernel: [  524.476447] dwc2 20980000.usb: new device is high-speed
Mar 17 16:41:35 Jha-RPi-USBDrive-01 kernel: [  524.488071] dwc2 20980000.usb: new address 18
Mar 17 16:41:36 Jha-RPi-USBDrive-01 kernel: [  524.496830] g_mass_storage gadget: high-speed config #1: Linux File-Backed Storage
Mar 17 16:42:43 Jha-RPi-USBDrive-01 kernel: [  591.934334] dwc2 20980000.usb: new device is high-speed
Mar 17 16:42:43 Jha-RPi-USBDrive-01 kernel: [  591.946505] dwc2 20980000.usb: new address 19
Mar 17 16:44:40 Jha-RPi-USBDrive-01 kernel: [  709.023131] dwc2 20980000.usb: new device is high-speed
Mar 17 16:44:40 Jha-RPi-USBDrive-01 kernel: [  709.034635] dwc2 20980000.usb: new address 20
Mar 17 17:04:04 Jha-RPi-USBDrive-01 kernel: [ 1873.382496] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:04 Jha-RPi-USBDrive-01 kernel: [ 1873.394712] dwc2 20980000.usb: new address 21
Mar 17 17:04:11 Jha-RPi-USBDrive-01 kernel: [ 1880.407365] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:11 Jha-RPi-USBDrive-01 kernel: [ 1880.419197] dwc2 20980000.usb: new address 22
Mar 17 17:04:14 Jha-RPi-USBDrive-01 kernel: [ 1883.434226] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:14 Jha-RPi-USBDrive-01 kernel: [ 1883.446854] dwc2 20980000.usb: new address 23
Mar 17 17:04:17 Jha-RPi-USBDrive-01 kernel: [ 1885.671076] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:17 Jha-RPi-USBDrive-01 kernel: [ 1885.682040] dwc2 20980000.usb: new address 24

Here you could see Pi has initially detected and not later on.
Moreover I am using separate cable for power supply and data transfer because if I use only one USB cable connected in Data port it will only provide power supply will not work to transfer data.

[Linux_01]

And I am using a file to save the data not partition for testing purpose I have 8GB memory card:

Code: Select all

root@Jha-RPi-USBDrive-01:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.3G  3.2G  3.7G  47% /
devtmpfs        213M     0  213M   0% /dev
tmpfs           217M     0  217M   0% /dev/shm
tmpfs           217M  3.3M  214M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           217M     0  217M   0% /sys/fs/cgroup
/dev/mmcblk0p1   44M   23M   21M  52% /boot
/dev/loop0      197M   13M  185M   7% /mnt/usb_share
tmpfs            44M     0   44M   0% /run/user/1000

How can I again partition my root

[thagrol]

When I connect the Pi as USB thumb drive for the 1st time to the MAC book, it detects and able to transfer the data.

If I disconnect the Pi from MAC book without ejecting it an dthen reconnect MAC doesn't detect where as at Pi end logs shows it detects the reinsertion but cannot map:

Code: Select all

Mar 17 16:41:35 Jha-RPi-USBDrive-01 kernel: [  524.476447] dwc2 20980000.usb: new device is high-speed
Mar 17 16:41:35 Jha-RPi-USBDrive-01 kernel: [  524.488071] dwc2 20980000.usb: new address 18
Mar 17 16:41:36 Jha-RPi-USBDrive-01 kernel: [  524.496830] g_mass_storage gadget: high-speed config #1: Linux File-Backed Storage
Mar 17 16:42:43 Jha-RPi-USBDrive-01 kernel: [  591.934334] dwc2 20980000.usb: new device is high-speed
Mar 17 16:42:43 Jha-RPi-USBDrive-01 kernel: [  591.946505] dwc2 20980000.usb: new address 19
Mar 17 16:44:40 Jha-RPi-USBDrive-01 kernel: [  709.023131] dwc2 20980000.usb: new device is high-speed
Mar 17 16:44:40 Jha-RPi-USBDrive-01 kernel: [  709.034635] dwc2 20980000.usb: new address 20
Mar 17 17:04:04 Jha-RPi-USBDrive-01 kernel: [ 1873.382496] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:04 Jha-RPi-USBDrive-01 kernel: [ 1873.394712] dwc2 20980000.usb: new address 21
Mar 17 17:04:11 Jha-RPi-USBDrive-01 kernel: [ 1880.407365] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:11 Jha-RPi-USBDrive-01 kernel: [ 1880.419197] dwc2 20980000.usb: new address 22
Mar 17 17:04:14 Jha-RPi-USBDrive-01 kernel: [ 1883.434226] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:14 Jha-RPi-USBDrive-01 kernel: [ 1883.446854] dwc2 20980000.usb: new address 23
Mar 17 17:04:17 Jha-RPi-USBDrive-01 kernel: [ 1885.671076] dwc2 20980000.usb: new device is high-speed
Mar 17 17:04:17 Jha-RPi-USBDrive-01 kernel: [ 1885.682040] dwc2 20980000.usb: new address 24

Here you could see Pi has initially detected and not later on.

I still suspect this is a Mac OS related problem so can't offer further help.

Moreover I am using separate cable for power supply and data transfer because if I use only one USB cable connected in Data port it will only provide power supply will not work to transfer data.

Never had a problem doing that. Check the cable and make sure that when using a single cable it's plugged into the data port on the zero.

[thagrol]

And I am using a file to save the data not partition for testing purpose I have 8GB memory card:

Code: Select all

root@Jha-RPi-USBDrive-01:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.3G  3.2G  3.7G  47% /
devtmpfs        213M     0  213M   0% /dev
tmpfs           217M     0  217M   0% /dev/shm
tmpfs           217M  3.3M  214M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           217M     0  217M   0% /sys/fs/cgroup
/dev/mmcblk0p1   44M   23M   21M  52% /boot
/dev/loop0      197M   13M  185M   7% /mnt/usb_share
tmpfs            44M     0   44M   0% /run/user/1000

How can I again partition my root

You can't re-partition it while it's running so you'll need another linux box or another SD card to boot from and a card reader.

Once you have that, gparted is probably the easiest way but it needs the desktop.

Run gparted (you'll need to be root or use sudo) . Select the card you want to change. Reduce the size of teh second partition (assming a direct install not noobs). Create a new partition in the free space.

How you fornat it depends on how you're going to pass it to g_mass_storage. If you intend to pass the partition to the UBS host device, I've found it best to format it via the USB host device.

If you want to stick with using a file as backing store for the USB gadget, format the new partition via gparted when you create it (ext4 would probably be best).

Either way, if you need it mounted on the zero you'll need to create a mount point and add an entry to /etc/fstab.

[Linux_01]

Today I checked connecting only one USB cable to the pi and MAC and it works.
Pi is able to get the power as well as able to transfer the data.
Also able to reconnect and mount the USB Drive at MAC after disconnecting abruptly with following fsck message in the logs:

Code: Select all

Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   24.927265] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   24.983265] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   25.039293] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.094999] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.150778] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.206651] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.297716] dwc2 20980000.usb: new address 7
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.323644] g_mass_storage gadget: high-speed config #1: Linux File-Backed Storage
Mar 19 11:17:22 Jha-RPi-USBDrive-01 kernel: [   26.070905] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Mar 19 11:17:22 Jha-RPi-USBDrive-01 kernel: [   26.692068] FAT-fs (loop0): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

I have formatted files system as vfat and file storage name is "piusb.bin"
Tried executing:

Code: Select all

fsck.vfat -a /piusb.bin

With no luck.
Please suggest how shall I go further?

[thagrol]

Today I checked connecting only one USB cable to the pi and MAC and it works.
Pi is able to get the power as well as able to transfer the data.
Also able to reconnect and mount the USB Drive at MAC after disconnecting abruptly with following fsck message in the logs:

Code: Select all

Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   24.927265] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   24.983265] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:20 Jha-RPi-USBDrive-01 kernel: [   25.039293] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.094999] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.150778] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.206651] dwc2 20980000.usb: new device is high-speed
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.297716] dwc2 20980000.usb: new address 7
Mar 19 11:17:21 Jha-RPi-USBDrive-01 kernel: [   25.323644] g_mass_storage gadget: high-speed config #1: Linux File-Backed Storage
Mar 19 11:17:22 Jha-RPi-USBDrive-01 kernel: [   26.070905] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Mar 19 11:17:22 Jha-RPi-USBDrive-01 kernel: [   26.692068] FAT-fs (loop0): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

I have formatted files system as vfat and file storage name is "piusb.bin"
Tried executing:

Code: Select all

fsck.vfat -a /piusb.bin

With no luck.
Please suggest how shall I go further?

Try

Code: Select all

sudo fsck.vfat /dev/loop0

If that retruns an error rather than performing the check, post the error message. Alternatively check it from the Mac side of things.

[PiGraham]

Moreover I am using separate cable for power supply and data transfer because if I use only one USB cable connected in Data port it will only provide power supply will not work to transfer data.

Two cables could be a problem. If the Pi0 remains powered as you disconnect and reconnect to a host maybe it fails to reinitialise as it would on a single connection with data and power. IDK, just a thought.

You may just need a good USB cable. Some charge cables don't have data connections and some data cables may have thin power wires that drop too much voltage for the Pi to run.

I can run a Pi4 in gadget mode on a single USB C cable than came with mit phone, but a cable that came with a power bank doesn't work. The Pi4 is not recognised as a USB device in that case..