kryptedbbkick
Posts: 1
Joined: Fri Jul 12, 2019 8:48 am

Pi Zero W as USB dongle for a Tesla

Fri Jul 12, 2019 9:02 am

Ok so I've been trying to wrap my head around this and get it working. Basically im trying to get a pi zero W to act as a USB dongle. Now when i say that i do not mean get powered through the usb and ssh into it. Thats a plus but the least of my concern. I want to be able to plug it in and have it boot up and lets say in windows show up as a usb i can then click to open and store files within. I've followed this tut https://www.raspberrypi.org/magpi/pi-ze ... ash-drive/ but i have had no success as of right now. I can get it up and running it shows that i have an 8gb storage container but when plugging it into a pc it dose not show up.

My main goal is to have this plugged into my tesla. My tesla will send the camera videos to the "PiUSB" and then i can use the pi's hotspot to download these videos to my phone or have the auto send using something like "Seafile" or "Samba" and copy them over to my computer without ever having to touch it. Can someone who might have worked on a project like this give me a hand?

--Edit--
I know i specified "Tesla" but honestly making it pop up in Windows as a "USB" is basically what im trying to get.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Fri Jul 12, 2019 11:19 am

That tutorial is both flawed and works in the wrong direction (server to Pi to USB host not USB to Pi to sever which you want).

I've had success multiple time like this:
  1. Add

    Code: Select all

    dtoverlay=dwc2
    to /boot/config.txt
  2. Add

    Code: Select all

    modprobe g_mass_storage file=[...]
    to /etc/rc.local above the "exit 0" line. Replace [...] with the full parth to the file or device node for the partition you're using as backing storage.
Now, as to what you want to do. I'm afraid it's going to be difficult or (almost) impossible. This is going to get technical so bear with me.

All the Pi will see is requests from the USB host to read and/or write N bytes at address A in the shared storage. It does not know which file that relates to, whether there are more writes to that file to come, or what filesystem is in use.

There is no mechanism to comunicate the nature of the changes from the USB host to tyhe OS on the Pi so even if you have the shared storage mounted on the Pi you won't see the changes. Likewise in the other direction.

It gets worse: both ends of the USB link use some level of caching so it's possible for one side to get data the other thinks has been deleted or changed. If you're trying to write from both sides things get messy very quickly.

You can update one side on what the other has done by unmounting and remounting the shared storage. The trick is knowing when it's safe to do so. It can be, kinda, done in that tutorial because the TV doesn't write to the shared storage and as you've copied the files to the Pi you know when it's safe from its end of the link. However it takes no account of how the TV will handle having the storage removed while in the middle of playback.

In your case, you have no way to know what the USB host is doing and whether it has finished writing to a file or whether there is more data to come.

Refreshing the Pi's view by unmount and mount will update what it sees but that view will rapidly be out of date. You can copy those files but as things can change under your feet you'll get incomplete or corrupt data. This applies even to old files due to house keeping in the filesystem and wear leveling on flash storage (chunks of the file can get moved around on the disc)

In conclusion: that tutorial is wrong for your use case. You can get close to what you want but not exactly there.
This space unintentionally left blank.

zwieblum
Posts: 10
Joined: Sun Jul 07, 2019 6:55 pm

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 8:32 am

On the other hand you might get around with a little trick: measure the io stats of the assigned device, if there is no traffic for e.g. 10 secs, then mount the device readonly, copy/rsync/whatever the contents to a different location, unmount the device again. This works as long as your gadgets filesystem is mounted in sync mode. Another trick is to use a filesystem that supports snapshots. before opening the image, make a snapshot and mount that readonly. if done with copying, destroy the snapshot. Anyway, you have to make sure that the images/videos are not corrupt before copying.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 4:36 pm

zwieblum wrote:
Sat Jul 13, 2019 8:32 am
On the other hand you might get around with a little trick: measure the io stats of the assigned device, if there is no traffic for e.g. 10 secs, then mount the device readonly, copy/rsync/whatever the contents to a different location, unmount the device again. This works as long as your gadgets filesystem is mounted in sync mode.
Which very much depends on what the USB host does and we have no control over that. Plus if the USB host is continuously writing (these are video camera streams) it's unlikely to be idle for 10 seconds.

Using the sync mount option on the Pi side isn't going to help.
Another trick is to use a filesystem that supports snapshots. before opening the image, make a snapshot and mount that readonly. if done with copying, destroy the snapshot. Anyway, you have to make sure that the images/videos are not corrupt before copying.
Fine in theory but requires that the USB host support said filesystem. Which it apparently doesn't what littel evidence I've seen suggests FAT only.

And stil doesn't address the risk of incomplete or corrupt files.
This space unintentionally left blank.

zwieblum
Posts: 10
Joined: Sun Jul 07, 2019 6:55 pm

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 6:53 pm

USB host does not know on which filesystem the file that is presented as storage devise is on.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 7:10 pm

zwieblum wrote:
Sat Jul 13, 2019 6:53 pm
USB host does not know on which filesystem the file that is presented as storage devise is on.
Yes it does. The host (the device acting as the USB master i.e. the tesla) knows what the filesystem is and how to write correctly formatted data to it.

Well unless it's writing raw data to the partition. Which it won't be as that is not very user friendly when it come to getting at the data from another device.

On the USB slave (the Pi zero) the g_mass_storage module neither know or cares what filesystem is in use all it does is transfer data based on block level commands received. That data could be for a file, it could be meta data. It doesn't know.

If you want to access that data on linux (whether on the zero or over a network) the linux box has to understand the filesystem too.

In general, block devices know nothing about filesystems or files. That's the job of something higher up the chain.

I guess you could have misunderstood what I meant by "USB host".

About the only way I can think of to get simultaneous access to the shared storage and avoid incomplete or corrupted files is to use a cluster aware filesystem (and possiblt iSCI to connect to it). All devices, including the device providing the storage have to understand both the filesystem and the mechanism used to access it.

Good luck getting that to work on a consumer device (a telsa). FAT32? Probably. exFAT? Maybe. Anything else is very unlikely.
This space unintentionally left blank.

zwieblum
Posts: 10
Joined: Sun Jul 07, 2019 6:55 pm

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 7:42 pm

Well, IMO in this example the Tesla is the USB-host, RPi is usb-device. RPi handles a bunch of byted over to host as storage device, e.g. by using

Code: Select all

modprobe g_mass_storage file=/a/file/of/1GB.img
/a/file/of/1GB.img now lives on a filesystem that supports snapshots. the contents of /a/file/of/1GB.img is e.g. raw FAT32 or MSDOS partitiontable with some partitions. Now you can take a snapshot of /a/file/of/1GB.img, mount it readonly as an image and analyse it. It helps to analyze the partition table (if preasent) and extract a propper offset before mounting :). When you are one, discard the snapshot.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 8:26 pm

zwieblum wrote:
Sat Jul 13, 2019 7:42 pm
Well, IMO in this example the Tesla is the USB-host, RPi is usb-device.
So we're both on the same page there.
RPi handles a bunch of byted over to host as storage device, e.g. by using

Code: Select all

modprobe g_mass_storage file=/a/file/of/1GB.img
/a/file/of/1GB.img now lives on a filesystem that supports snapshots. the contents of /a/file/of/1GB.img is e.g. raw FAT32 or MSDOS partitiontable with some partitions. Now you can take a snapshot of /a/file/of/1GB.img, mount it readonly as an image and analyse it. It helps to analyze the partition table (if preasent) and extract a propper offset before mounting :). When you are one, discard the snapshot.
So if I understand what you're proposing correctly:
  • You suggest using a file on some unspecified partition to act as the backing storage for g_mass_storage rather than a partition.
  • THIS partition uses a filesystem that supports snapshots.
  • The USB host doesn't see this partition only the file masquerading as a partition via g_mass_storage (correct). That wasn't what you said first tiMe around, though it may have been what you meant.
  • You take a snapshot of the file or filesystem containg the file that is used for shared storage and mount the snapshot before copying the data and delete the snapshot afterwards.
  • You suggested using the sync mount option but on which filesystem? The snapshotable one on the Pi? The one exported to the USB host?
I don't really see the advantage in the added complexity. Sure it stops the Pi end from having the data changed under its feet but:
  • It doesn't address the issue of incomplete files in the shared storage.
  • Sync on the Pi's filesystem alone isn't going to help. With sync, writes made by the Pi (whether by g_mass_storage or not) should be written directly to the storage but you still have no idea what the USB host is doing and whether it has finished writing.
  • My comment about waiting for storgae to be idle still applies.
  • You're going to need a lot of free space on the Pi side of the link in order to be able to make the snapshot. How much depends on the filesystem in use, how many changes have been to it and other factors (can the chosen fs cope with partial changes to a file or does it need to keep multiple version of the entire thing? Does it even keep anything or does it just lock the entie FS and copy the file when a snapshot is requested?).
  • There are going to be performance issues with snapshots and snapshot capable filesystems. You don't get them for free.
  • What if you actually want to export a partition or entire block device? In this case you can't use a snapshot capble FS unless the host device supports it.
I admit I assumed a partition or device would be shared. Or that if using a file for shared storage that file would be treated as a block device and formatted with a snapshotable FS. I did not consider that you meant the filesystem containing the file that contains the shared storage. I mean why would I? That's an inefficient way of doing things.
This space unintentionally left blank.

zwieblum
Posts: 10
Joined: Sun Jul 07, 2019 6:55 pm

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 9:11 pm

Define "inefficient". If you want to access one filesystem accessable from 2 computers as actual filesystems, you have to live with what you can get. Snapshots are practically free on filesystems that support snapshops (copy-on-write), no runtime penalty when taking a snapshot. You can run out of diskspace if you do not provide a bit more than double the space of your image. Anyway, memory is cheep nowadays.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 10:23 pm

We're getting off topic but hey.
zwieblum wrote:
Sat Jul 13, 2019 9:11 pm
Define "inefficient".
Sure: poor use of available space and reduced performance FS.
If you want to access one filesystem accessable from 2 computers as actual filesystems, you have to live with what you can get.
I never said otherwise. In the particular case of g_mass_storage I'd always go with unmounting the side that isn't making the change, changing stuff then remounting the unmounted side. That way you know the write buffers have been flushed and written to disc.

Where possible, I'd only have it mounted on one side at a time too.
Snapshots are practically free on filesystems that support snapshops (copy-on-write), no runtime penalty when taking a snapshot. You can run out of diskspace if you do not provide a bit more than double the space of your image.
Taking the snapshot might be (almost) free. The trouble with copy on write is that the next write is going to be slow as the data has to be copied first.

This is compounded by most CoW filesystem only implementing this at the file level. So the entire shared storage .img file has to be copied before it can be written to. For a multi GB storage file that's going to take a while.

And don't forget there are things that wrtie to disc when you're not expecting it. Last accessed time for one.

Plus I've no idea if either g_mass_storage or the USB host can cope with being detached from the storage file then reattached after the copy. Even if they can I'd expect to see timeout errors on the USB host end.
Anyway, memory is cheep nowadays.
Do you mean disc/storage? Whether memory/RAM is cheap or not doesn't matter with a Pi as its memory cannot be upgraded.
This space unintentionally left blank.

zwieblum
Posts: 10
Joined: Sun Jul 07, 2019 6:55 pm

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 10:36 pm

CoW works on blocks, not files. And it's the bauty of snapshots that they are transparent to conumers.

User avatar
thagrol
Posts: 1619
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pi Zero W as USB dongle for a Tesla

Sat Jul 13, 2019 11:57 pm

zwieblum wrote:
Sat Jul 13, 2019 10:36 pm
CoW works on blocks, not files. And it's the bauty of snapshots that they are transparent to conumers.
Fair enough. I stand corrected though I still have doubts of the benefit of the added complexity.
This space unintentionally left blank.

Return to “Other projects”