User avatar
jbeale
Posts: 3884
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Pico as USB memory device ?

Sat May 22, 2021 6:26 pm

Is there any example code which when run on a Pico, causes the Pico to appear to an external USB host as a simple USB memory stick?

In some cases it would be convenient to present some live sensor data as a dynamically updated file on the Pico, that the host could access and read periodically as if it was a normal file on a simple USB memory stick, without having to change existing software on the host side.

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sat May 22, 2021 7:31 pm

CircuitPython does that automatically but I haven't seen any equivalent for MicroPython.

Might be worth playing with that to check you can make it work, can get round any caching the host may be doing. I believe CircuitPython USB is dynamically configurable so you may be able to get the disk to disappear and reappear which may help things.

Two other options - USB networking with a HTTP or FTP server to deliver data as files. A FUSE-based file system which can create virtual files on a host Pi system by communicating with the Pico via serial over USB. No implementations of either yet.

User avatar
jbeale
Posts: 3884
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 3:34 am

I was able to try the CircuitPython example that logs data to the internal flash memory, but it doesn't work exactly the way I had in mind.
In particular, the host PC mounts the board's filesystem in "Read Only" mode and it can read the file exactly once. Either on the host side or the client side (or both?) apparently "Read Only" means the file will never change so it is read once and never again, even while the Pico continues to write new data. The only way to read the new data is (apparently) to physically unplug and re-plug the board, which of course also resets it.

I tried closing the file, remounting as R/W to the host, then remounting as RO to host (R/W to Pico) and re-opening the file (example below), but this did not work even once. It may be that storage.remount() can be called only one time per session.

Code: Select all

... open file and write stuff, then ...

fp.close()
storage.remount("/", True) # host sees device filesystem as R/W
time.sleep(1)
storage.remount("/", False) # host sees device filesystem as Read-Only
fp = open("/temperature.txt", "a")
I guess the conclusion is that the functionality I want (Pico periodically updates a file, which is also periodically read via USB from host) is simply not possible, at least with the current version of CircuitPython. I wonder if it's possible in C/C++ for example.
Last edited by jbeale on Sat Jun 05, 2021 3:37 am, edited 2 times in total.

cleverca22
Posts: 3979
Joined: Sat Aug 18, 2012 2:33 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 3:37 am

jbeale wrote:
Sat Jun 05, 2021 3:34 am
In particular, the host PC mounts the board's filesystem in "Read Only" mode and it can read the file exactly once. Either on the host side or the client side (or both?) apparently "Read Only" means the file will never change so it is read once and never again, even while the Pico continues to write new data. The only way to read the new data is (apparently) to physically unplug and re-plug the board, which of course also resets it.
when emulating a normal mass-storage device, the host computer assumes it has exclusive control of the flash memory
and it can cache things it has read, because its not expecting the contents to change on their own

usb camera and phone stuff has had the exact same limitation, the device cant use the partition while it is mounted by another system

the MTP protocol was made to solve that, but i dont know the details of implementing it

User avatar
jbeale
Posts: 3884
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 3:42 am

cleverca22 wrote:
Sat Jun 05, 2021 3:37 am
when emulating a normal mass-storage device, the host computer assumes it has exclusive control of the flash memory
and it can cache things it has read, because its not expecting the contents to change on their own
usb camera and phone stuff has had the exact same limitation, the device cant use the partition while it is mounted by another system
the MTP protocol was made to solve that, but i dont know the details of implementing it
Ah, ok, I did not realize this although I suppose that makes sense.

I wonder if there is a way to emulate a network-attached storage device over USB. I think the usual case there is that other devices may be expected to write to the filesystem at any time. Maybe existing code to support a NAS device is too heavyweight for a Pico?

cleverca22
Posts: 3979
Joined: Sat Aug 18, 2012 2:33 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 3:47 am

jbeale wrote:
Sat Jun 05, 2021 3:42 am
Ah, ok, I did not realize this although I suppose that makes sense.

I wonder if there is a way to emulate a network-attached storage device over USB. I think the usual case there is that other devices may be expected to write to the filesystem at any time. Maybe existing code to support a NAS device is too heavyweight for a Pico?
viewtopic.php?p=1835857#p1835857

once you have tcp/udp, you can then implement any other protocol you want

User avatar
thagrol
Posts: 5265
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 11:17 am

jbeale wrote:
Sat Jun 05, 2021 3:42 am
cleverca22 wrote:
Sat Jun 05, 2021 3:37 am
when emulating a normal mass-storage device, the host computer assumes it has exclusive control of the flash memory
and it can cache things it has read, because its not expecting the contents to change on their own
usb camera and phone stuff has had the exact same limitation, the device cant use the partition while it is mounted by another system
the MTP protocol was made to solve that, but i dont know the details of implementing it
Ah, ok, I did not realize this although I suppose that makes sense.

I wonder if there is a way to emulate a network-attached storage device over USB. I think the usual case there is that other devices may be expected to write to the filesystem at any time. Maybe existing code to support a NAS device is too heavyweight for a Pico?
It's the same issue as with the mass sotrage gadget on a Pi.

If you don't want to go the NAS over USB ethernet route, consider this:
  • Have the pico present to the PC as a mass storage device with removable storage.
  • Keep two copies of your data: a static one exposed via mass storage to the PC and a live copy that's being updated.
  • When you want to update the static copy, send an ejected signal to the PC, update the static copy then "re-insert". Or use some other method of triggering a reread on the PC (disable and reenable the mass storage function maybe)
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

fanoush
Posts: 786
Joined: Mon Feb 27, 2012 2:37 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 11:34 am

cleverca22 wrote:
Sat Jun 05, 2021 3:37 am

when emulating a normal mass-storage device, the host computer assumes it has exclusive control of the flash memory
and it can cache things it has read, because its not expecting the contents to change on their own
yes but still it is possible and I've seen systems doing such communication via usb storage, not via files but direct block access (e.g. 'dd' in linux). maybe with some explicit cache flushing or direct io/uncached access it could work via files too (O_DIRECT open flag).

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sat Jun 05, 2021 12:32 pm

The problem is, as noted, caching. No matter how access is achieved, stream or block, one has to force the PC or host to realise that what it has read has changed, that it must read the data from the device not use the cached data.

For MSD this is normally done, as 'thagrol' says, by the device removing or ejecting itself, advertising itself as removable when mounted. That's the only reliable way I know to facilitate the device clearing the host's cache.

Setting up as a NAS probably won't work any better because it has the same issues in my experience; though they tend not to be noticed because using a NAS is a host driven experience. You put an MP3 on a NAS, someone else overwrites that with a different MP3 of the same name, you will hear your MP3 not theirs. Until an F5 or whatever 'flush cache' is done on the host.

It can appear it does work, play your MP3 and you get yours, the second play it gets theirs, each play does get the latest MP3 of the same name they have uploaded. But that's usually because playing filled the cache so it is forced to read the device because the data it wants is no longer cached. It's effectively flushing the cache by filling it with stuff it can't use.

So you are not likely to do any better than using the in-built CircuitPython MSD capabilities.

I suspect the problem with your testing is that 'mount' and 'remount' do not do what is required, nor 'umount' either. It seems to me you need to use 'enable_usb_drive' and 'disable_usb_drive' to affect what the host is seeing. Enable to present it and have the host mount it, disable to eject it and invalidate any cached data.

That's probably also why I couldn't get my Pico to work as an MSD with my Digital Picture Frames. If I can figure out why I can no longer build CircuitPython from source I'll check to see if that is the fix for my case.

fanoush
Posts: 786
Joined: Mon Feb 27, 2012 2:37 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 1:34 pm

should work with O_DIRECT, here is similar case https://unix.stackexchange.com/question ... e-in-linux
However the read must be block aligned, not sure what is the smallest block size usb storage block device can have, 515 bytes maybe?

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sat Jun 05, 2021 2:34 pm

The problem AIUI with O_DIRECT is it doesn't have to be supported nor honoured. It would probably be necessary to write ones own code to use it so isn't a generic solution, is likely platform specific.

Ideally one wants a 'status.txt' or 'graph.jpg' to show on an MSD on any platform and being able to use that however any file would be used with existing tools, the user getting the latest data a device has put in that. That simply doesn't work without an eject and remount when the data changes..

cleverca22
Posts: 3979
Joined: Sat Aug 18, 2012 2:33 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 3:35 pm

thats what MTP is for

MTP is a FILESYSTEM over use, rather then a block device

so you just ask the pico for the new contents of a given file, without any caching problems

User avatar
thagrol
Posts: 5265
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 4:12 pm

cleverca22 wrote:
Sat Jun 05, 2021 3:35 pm
thats what MTP is for

MTP is a FILESYSTEM over use, rather then a block device

so you just ask the pico for the new contents of a given file, without any caching problems
Isn't the issue about pushing changes from the pico to the PC? Can MTP (or PTP) do that? (not trolling, genuinely want to know)
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

cleverca22
Posts: 3979
Joined: Sat Aug 18, 2012 2:33 pm

Re: Pico as USB memory device ?

Sat Jun 05, 2021 4:21 pm

thagrol wrote:
Sat Jun 05, 2021 4:12 pm
cleverca22 wrote:
Sat Jun 05, 2021 3:35 pm
thats what MTP is for

MTP is a FILESYSTEM over use, rather then a block device

so you just ask the pico for the new contents of a given file, without any caching problems
Isn't the issue about pushing changes from the pico to the PC? Can MTP (or PTP) do that? (not trolling, genuinely want to know)
yeah, because MTP was made for things like a camera, where photos just appear on the device without warning, because of actions outside of the PC's control

User avatar
scruss
Posts: 4189
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 5:34 pm

hippy wrote:
Sat May 22, 2021 7:31 pm
CircuitPython does that automatically but I haven't seen any equivalent for MicroPython.
What, make live sensor data appear as a dynamically updated file? I thought it only provided a USB file system to the host, and not emulated FS access to readings. Which would be extremely cool if it did.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sat Jun 05, 2021 5:52 pm

MTP pulls entire files from a device without worrying how they are stored on that device, what file system, block size etc, its file system uses. But it's still not what it allows; it's how the host uses it which is crucial.

The host can always read the entire file, but it doesn't seem mandated that it has to, the protocol didn't seem to preclude a host from caching what has been read if it chooses to. And I am sure I saw 'bug reports' suggesting some hosts do cache, having to unplug and re-plug or even reboot to flush caches. I have rarely used MTP so can't test or report my own experiences.

Many cameras save their images, video and audio files with a timestamp or sequence number in their filename and the likelihood of a device presenting a new file using an existing filename seems unlikely. That makes it a very different scenario to the same file being repeatedly updated.

If I had to code an MTP Bridge I would add file caching to improve the 'let's immediately watch that again' experience. No one could stop me and I doubt any would complain as they would not likely encounter the issue where caching is detrimental. I would however have a cache bypass option because I know it can be detrimental.

Where the real difference is with MTP is in asking what's on the device, reading the directory, rather than with MSD when it assumes the directory it first read hasn't been altered, updated. So creating a new file with a new name, deleting the old, will work with MTP. Updating an existing file I wouldn't be so certain about.

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sat Jun 05, 2021 6:11 pm

scruss wrote:
Sat Jun 05, 2021 5:34 pm
I thought it only provided a USB file system to the host, and not emulated FS access to readings.
Yes, that was what I was meaning; CircuitPython will present an MSD drive and give access to files placed on the local file system through that MSD

Whether that MSD drive is presented or not is dynamically selectable. But not quite dynamically enough to be useful.

It seems the selection of whether it will be presented or not is merely setting or clearing flags as it boots (during boot.py), is presented or not at the end of that depending on flag settings. After that it cannot be changed on the fly. The MSD cannot pull itself, re-present itself for a fresh mounting on the host.

To me it's looking more and more likely the proper solution is to use a FUSE file system to present a file system which automatically clones the device data through some mechanism; a home grown and simpler MTP.

User avatar
thagrol
Posts: 5265
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pico as USB memory device ?

Sat Jun 05, 2021 10:58 pm

hippy wrote:
Sat Jun 05, 2021 6:11 pm
To me it's looking more and more likely the proper solution is to use a FUSE file system to present a file system which automatically clones the device data through some mechanism; a home grown and simpler MTP.
You're assuming a Linux host PC there. It may not be, the OP didn't say. You're also assuming it will be possible to install additional software on the USB host.

The advantage of using the mass storage protocol is that it's about as universal as it gets.

The disadvantage is that it seems never to have been deisgned for simultaneous access from two directly connected devices at a time compounded by caching on both devices.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Sun Jun 06, 2021 2:24 pm

thagrol wrote:
Sat Jun 05, 2021 10:58 pm
The advantage of using the mass storage protocol is that it's about as universal as it gets.
I entirely agree. But if that cannot be made to work there are few options but to look at alternatives which will, even if less than desirable.

But, with a bit of lateral thinking, I think I have effected pulling the MSD, re-presenting it.

CircuitPython doesn't provide a means to pull the drive, eject it, but a hard reset of a Pico will cause that. and we can use 'boot.py' to set how long after hard-reset it presents the MSD if that is needed. Thus we can present the MSD, pull it, re-present it, ad infinitum -

boot.py

Code: Select all

import time
time.sleep(10)
code.py

Code: Select all

import microcontroller, time
time.sleep(60)
microcontroller.reset()
That's working with my Raspberry Pi OS desktop; it presents itself for mounting 10 seconds after insertion, a minute later it disappears, 10 seconds after that it offers itself up for mounting again.

But it's not perfect ...

The hard reset sometimes causes the Raspberry Pi OS ACM-CDC drivers to break, as it does with MicroPython and Pico-SDK code, leading to freezing and/or needing to do a power-pull to reboot or actually shutdown.

It's not working with my Digital Picture Frames. Those appears to be recognising the MSD being presented and pulled but aren't displaying the image I have pre-loaded to CircuitPython's file system. It could be because it's presenting as a composite device not a pure MSD but maybe there's something about the drive's characteristics it doesn't like. They work with Pi Zero W's in MSD gadget mode so should work with a Pico.

It's not working with Windows yet but that could be a Windows and/or driver issue.

I also haven't tried live updating a file yet, haven't decided the best approach for that. I'm at a bit of a disadvantage there as I am not familiar with CircuitPython and how one interacts with MSD presented file systems.

User avatar
jbeale
Posts: 3884
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pico as USB memory device ?

Mon Jun 07, 2021 4:01 am

Wow, thanks everyone for the additional discussion of this issue. Indeed what I was hoping for was a general-purpose, non-OS-specific solution that would enable existing systems running on various legacy hardware to work "as-is" with a new sensor device, so long as they could access a file on external USB storage. I thought there might be a workaround to the caching problem by some USB disconnect / reconnect method, but I gather it has to be a hard reset of the USB device, and from this latest post I understand even that fairly drastic measure does not uniformly work.

User avatar
thagrol
Posts: 5265
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Pico as USB memory device ?

Mon Jun 07, 2021 8:35 am

jbeale wrote:
Mon Jun 07, 2021 4:01 am
various legacy hardware
If that legacy hardware is embeded systems it's possible they don't do any caching. Only one way to find out...
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Mon Jun 07, 2021 12:51 pm

I did some further testing and believe I have it working reliably under Raspberry Pi OS and probably other Linuxes. I avoid the ACM-CDC crashes by disabling CDC - if it doesn't create a CDC device the driver can't crash on CDC disappearance.

That of course makes it rather difficult to program things so an added switch or link to decide if CDC is enabled or not at power-up or hard-reset would make sense.

Tweaking CircuitPython to pull its USB and re-present it would be a better option than requiring a hard-reset but I wouldn't know where to start with that.

For my Digital Picture Frames I disabled CDC plus HID and MIDI, but still no joy, same seeing the presentation and resets, but not loading anything.

I am wondering if it being presented with a mere "1MB memory stick" is the issue. The DPF software may support a more limited range of devices than an OS would.

Ideally we would present a read-only virtual disk of whatever size we desire, with the contents of that formed from data hard-coded into the program, on the Flash file system or from RAM. The last might be essential if we don't want to wear out Flash prematurely. I am not sure if that's possible at present.

I think we are closer to "can be done" but am not convinced it would be appropriate for all desired use-cases

fanoush
Posts: 786
Joined: Mon Feb 27, 2012 2:37 pm

Re: Pico as USB memory device ?

Mon Jun 07, 2021 3:16 pm

as for cache flush - when simulating usb cd-rom there should be a way to signal that the tray is manually opened and media is changed. maybe usb floppy or card reader could also generate some media changed event. however it is debatable whether random OS can handle it better that complete usb storage removal

hippy
Posts: 10310
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Pico as USB memory device ?

Mon Jun 07, 2021 5:01 pm

Presenting as CD-ROM or something other than pure MSD is an interesting option and should work. But I am not convinced we could present as such because TinyUSB doesn't seem to support that and I doubt we can find anyone who would know how to update TinyUSB so it does.

It's the perennial "disillusionment problem" with computing; you can do anything you want, but only if someone else has provided the means or you are willing to put in the time and effort and have the skills to do it yourself

tannewt
Posts: 60
Joined: Tue Nov 17, 2020 1:14 am

Re: Pico as USB memory device ?

Mon Jun 07, 2021 10:55 pm

hippy wrote:
Mon Jun 07, 2021 5:01 pm
Presenting as CD-ROM or something other than pure MSD is an interesting option and should work. But I am not convinced we could present as such because TinyUSB doesn't seem to support that and I doubt we can find anyone who would know how to update TinyUSB so it does.

It's the perennial "disillusionment problem" with computing; you can do anything you want, but only if someone else has provided the means or you are willing to put in the time and effort and have the skills to do it yourself
I think you are think about "removable media". CircuitPython does actually track whether the drive is ejected. It does that to prevent re-automounted after an eject. The behavior of host OSes does vary a bit though. IIRC windows doesn't actually eject. Generally, we don't allow you to eject from CP code because there is no way to know if the host OS has flushed all writes. If you were read-only that could be ok though.

Return to “MicroPython”