USB image writer

[Paulaner]

Hi,

thanks for developing a new promising tool.

Is reading from USB and writing to backup image file in the Windows version already implemented or does it only support to write existing images to USB so far?

When I select my USB stick and click on "read" in Win 10 the tool crashes on my computer...

Best
Paulaner

[petecal]

Just downloaded usbimager_1.0.1-i686-win-gdi.zip
Run as Administrator.
"Access is denied.
Unable to unmount volumes on device."

First SD card is from my RPi, FAT32
Second SD card is FAT32 written with Windows 8.1

Running Windows 8.1 on desktop.

Pardon me if there was a solution published somewhere in the 8 pages of replyies.

[bzt]

thanks for developing a new promising tool.

Welcome!

Is reading from USB and writing to backup image file in the Windows version already implemented or does it only support to write existing images to USB so far?

Yes it is. all features are implemented in all versions.

When I select my USB stick and click on "read" in Win 10 the tool crashes on my computer...

That's a new! Please run with the command line flag "-v". A new window should appear with detailed logs. Please open an issue on gitlab and copy'n'paste those messages there. Or you could copy them here. I'll able to tell more when I see the logs.

Cheers,
bzt

[bzt]

Just downloaded usbimager_1.0.1-i686-win-gdi.zip
Run as Administrator.
"Access is denied.
Unable to unmount volumes on device."

You're not running it properly as Administrator maybe? You should not see "Access denied" message when running as Administrator.

My other guess would be that another application is actively using the device, that's why the WIN32 API can't dismount and lock the device. Or another application could have already locked the device.

EDIT: I've just noticed, why did you download the 1.0.1??? Download the latest, 1.0.5! If the problem still exists with the latest version, then run "USBImager.exe -v", open an issue on gitlab and paste the output there. I need to see the logs to tell you more.

Cheers,
bzt

[bzt]

When I select my USB stick and click on "read" in Win 10 the tool crashes on my computer...

I've found the problem! Believe it or not, this is actually a gcc bug! I've updated to the latest MSYS2 before I compiled 1.0.5 for Win, and I did not notice something went wrong (there was no warning during compilation).

I've narrowed the problem down and located that the crash was actually happening in stream.c at this line:

Code: Select all

wsprintfW((wchar_t*)str, L"%6" PRIu64 " %s / %" PRIu64 " %s%s%s", (ctx->readSize >> 20), lang[L_MIB], ...

Under WIN32, PRIu64 should be defined as "llu" by the POSIX headers. As it turned out wsprintfW with "llu" formating crashes! That's because gcc 10.2.0 incorrectly concatenates strings. I've replaced that line with

Code: Select all

wsprintfW((wchar_t*)str, L"%6u %s / %u %s%s%s", (unsigned int)(ctx->readSize >> 20), lang[L_MIB], ...

and now it compiles correctly with mingw-gcc 10.2.0. This workaround has a drawback though, the status line will overflow under Win with images above 4 Petabytes. Hopefully never gonna happen with common OS images (and even then only the displayed status text will overflow, the disk still will be written correctly.)

I made sure of it, that all "l" and "ll" formatting and PRIx string concatenation have been removed from wsprintfW arguments, and recompiled the Windows binary. Please download the zip again.

I've checked, the Linux version wasn't affected (however I've experienced other problems with gcc 10.2.0 under Linux, where it "forgot" to add the offset to the base address).

Cheers,
bzt

[bzt]

Dear All,

USBImager 1.0.5 is out! Not much changed.

Changelog
- Added support for ZStandard compressed disk images
- Added an initial version of ARM64 (AArch64) Linux binary
- No MacOS nor Ubuntu deb binary sadly

Unfortunately my machine that I used to run my VMs on proved again that computers work with smoke. When the smoke gets out, the computer stops working. I've tried to recompile 1.0.5 for all supported platforms, and it was almost successful: only Ubuntu deb and MacOS zip missing, I managed to compile for all the other platforms.

A fix for @Paulaner's problem also included, no crash under Windows any more.

Cheers,
bzt

[petecal]

EDIT: I've just noticed, why did you download the 1.0.1???

Because that was the first one I saw on gitlab.
I just got the correct one and will try but I hit something else that I need to experiment with.
I think you are right about another application.
It could be a problem with my Windows installation because if I just plug in an SD Card, even one with windows files on it I sometimes get a message about something is wrong with the card.
If I look at properties, tools, Check, it says no problems.
But I can tell it to scan anyway, it says Windows can't repair the drive while it is in use. But I haven't started anything that would be using the Card I just plugged in.
I need to do some more testing.

Thanks

[bzt]

Because that was the first one I saw on gitlab.

Okay, I've double checked, the link on README now points to 1.0.5 for sure.

I just got the correct one and will try but I hit something else that I need to experiment with.
I think you are right about another application.
It could be a problem with my Windows installation because if I just plug in an SD Card, even one with windows files on it I sometimes get a message about something is wrong with the card.
If I look at properties, tools, Check, it says no problems.
But I can tell it to scan anyway, it says Windows can't repair the drive while it is in use. But I haven't started anything that would be using the Card I just plugged in.

If it were a Linux, I'd say use lsof. Not sure about Windows, but maybe Process Monitor can show you which process has open files on the device. It has a utility called Filemon (Just a guess, I don't use Windows.)

Cheers,
bzt

[GlowInTheDark]

I use ProcExp (process explorer) for that (checking which process has a file open - that is keeping me from deleting that file).

ProcExp is also in the SysInternals suite. Very handy program.

[bzt]

Hi @Anitapatel,

I'm not exactly sure what was your intention with quoting that old post. But you should have used [ quote ] tags, btw.

Just to be clear, time flies by and things have changed since:

1. Although Raspberry Pi still recommends balenaEtcher on the homepage, they also have created Raspberry Pi Imager since that post. That is the official tool now. If you do not want to create backups and write custom images, just to create a bootable RaspiOS card, then that's the best tool for you.

2. USBImager got udisks2 support since, so it is handling privileged disk access exactly the same way as Pi Imager, balenaEtcher and Gnome Disks (the deb versions compiled with that at least, the zip version is an extremely minimal, dependency-free version that uses disk group permissions and requires nothing else than libX11).

And yes, USBImager is still UNIXy, does one job, and one job only: write/read compressed disk images. It doesn't require net connection in order to do that (paranoid users can relax, no ads nor telemetry possible without net).

And yes, USBImager is still the smallest of all, being as slim as possible. Size varies on platforms, but it's always less than 192K. For example under Windows you'll need to download:
- balenaEtcher installer is ca. 142 Megabytes (and this is just the installer!),
- Pi Imager is ca. 19 Megabytes, and
- USBImager is just 142 Kilobytes (about 134 times smaller than Pi Imager, and 1024 times smaller than etcher installer).

Cheers,
bzt

[Paulaner]

That's a new! Please run with the command line flag "-v". A new window should appear with detailed logs. Please open an issue on gitlab and copy'n'paste those messages there. Or you could copy them here. I'll able to tell more when I see the logs.

Cheers,
bzt

I have no explanation, but backup is working now. Maybe I had to many devices on the USB hub...

Thank you very much

Edit: I just saw that you updated to a newer version. It's working

[bzt]

Hi,

Good news, the USBImager repo just reached 100 stars! I'd like to say thanks to all of its users, for all the precious feedback and all the support! Special thanks to all the contributors involved, I couldn't have done USBImager without you!

Next goal: have 500 stars and then take on the world!!!


Cheers,
bzt

[CaptainMidnight]

Hi,

Good news, the USBImager repo just reached 100 stars! I'd like to say thanks to all of its users, for all the precious feedback and all the support!

...... thank you, for creating a multiplatform imaging/backup tool and being open minded to potential project development requests.

[DarkElvenAngel]

Hi,

Good news, the USBImager repo just reached 100 stars! I'd like to say thanks to all of its users, for all the precious feedback and all the support! Special thanks to all the contributors involved, I couldn't have done USBImager without you!

Next goal: have 500 stars and then take on the world!!!


Cheers,
bzt

Awesome! Great to hear, you already have my star. Btw nice choice on Purple Tentacle now I have to dig out and play Day of the Tentacle.

[kerry_s]

no ubuntu desktop version yet.
i normally just use disks, but it would be nice to have a dedicated app.

[bzt]

Awesome! Great to hear, you already have my star. Btw nice choice on Purple Tentacle now I have to dig out and play Day of the Tentacle.

You're welcome and thanks! Do that, it's a really awesome game! I played it many many times when I was young, and I've recently played it through again with my boy, that's what gave me the idea. I wish there were more ScummVM games like DoTT!

no ubuntu desktop version yet.
i normally just use disks, but it would be nice to have a dedicated app.

There's a deb version for 1.0.4.

Sadly my machine that I used to run VMs to compile packages went out of order, and since then I can't compile all variants. Although I'm able to compile the deb on my current PC, but as it's not debian based, I have considerably newer versions of the shared libs meaning it's not possible to install the resulting deb on current Ubuntu (glibc is too old for one). That's why I haven't uploaded the deb variant. But the only difference between 1.0.4 and 1.0.5 is the zstd support, everything else is the same.

If someone could compile a 1.0.5 deb version for me under Ubuntu, I would be glad to add it to the binaries repo.

Code: Select all

$ USE_LIBUI=yes USE_UDISKS2=yes make all deb

Cheers,
bzt

[donmerch]

I have a 16GB sd card from my Raspberry Pi 4B that I want to archive and save as a file to be used later to make a new card in case I crash my present card. I've installed your version 1.0.5 on my Pi4B. I put the card I want to read into an sd card reader and plugged into the USB jack. I then opened your program and selected the device "sda [15.8 GiB] Multi Flash Reader". I then click on Read and I get this error: "GDBus.Error:org.freedesktop.UDisks2.Error.NotAuthorizedCanObtain: Not authorized to perform operation" so I don't know how to proceed with that.

I also have a MacBook Pro with macOS version Catalina 10.15.7. I downloaded the mac version of your software and got by the security issue of running the program and got it to run. I select "disk2 Generic USB3.0 Card Reader" and then click on Read. I get the message "Permission Denied". I followed your suggestion and added the USBImager to the Full Disk Access list but still get this error message. I then tried your other suggestion to run it in terminal:

Code: Select all

donmerchant@Dons-MacBook-Pro ~ % sudo /Applications/USBImager.app/Contents/MacOS/usbimager
Password:
2020-11-27 09:34:00.052 usbimager[8678:363309] get 0x7fff87a6f148

It then just sits there showing that bottom line. Is it doing anything? I waited a while but did not see anything happening so I pressed control c and got back to the prompt.

Help!!

[donmerch]

I was using remote desktop on my mac to the Pi4B. I just figured out that if I run the program on the Pi4B itself that it does seem to work. I'll keep you posted as to the result of that. Still curious as to why it does not work on my mac though.

[bzt]

Hi,

I have a 16GB sd card from my Raspberry Pi 4B that I want to archive and save as a file to be used later to make a new card in case I crash my present card. I've installed your version 1.0.5 on my Pi4B. I put the card I want to read into an sd card reader and plugged into the USB jack. I then opened your program and selected the device "sda [15.8 GiB] Multi Flash Reader". I then click on Read and I get this error: "GDBus.Error:org.freedesktop.UDisks2.Error.NotAuthorizedCanObtain: Not authorized to perform operation" so I don't know how to proceed with that.

That UDisks2 error is interesting, because in lack of privilege UDisks2 should prompt you for authorization instead of showing an error message. Try the X11 version of USBImager. That has setgid bit set for disk group, meaning you should be able to access any disk without additional authorization.

I also have a MacBook Pro with macOS version Catalina 10.15.7. I downloaded the mac version of your software and got by the security issue of running the program and got it to run. I select "disk2 Generic USB3.0 Card Reader" and then click on Read. I get the message "Permission Denied". I followed your suggestion and added the USBImager to the Full Disk Access list but still get this error message.

That's strange. Are you sure you have added Full Disk Access properly? Have you tried to reboot and see if USBImager is still in the list? Sorry, I'm just guessing here, because I don't have Catalina.

I then tried your other suggestion to run it in terminal:

Code: Select all

donmerchant@Dons-MacBook-Pro ~ % sudo /Applications/USBImager.app/Contents/MacOS/usbimager
Password:
2020-11-27 09:34:00.052 usbimager[8678:363309] get 0x7fff87a6f148

It then just sits there showing that bottom line. Is it doing anything? I waited a while but did not see anything happening so I pressed control c and got back to the prompt.

Sadly this is a known issue. I've tried to fix it many times, but the problem is, one of the core functions, CFRunLoopRun() freezes and never returns. I've asked people to try out some alternatives, you can find those here, but I got no response in that matter, so I don't know if they worked. If you could give those alternatives a try, that would be awesome.

I know it's inconvenient, but there's a workaround: manually umount (but don't eject) the device from a Terminal using "diskutil umountDisk /dev/diskX", after that USBImager will work perfectly.

ps.: hopefully within a few months I'll be able to buy a new machine and restore my VMs, so that I'll be able to compile and test MacOS versions again. In the meantime I'll help you any way I can. Please don't hesitate to open an issue on gitlab!

Cheers,
bzt

[rrobinet]

Thanks so much for this great program!
But I can't run on Mac OSX Big Sur 11.0.1:

rob@Robs-MBP-New:/Users/rob> sudo diskutil unmountDisk /dev/disk2
Unmount of all volumes on disk2 was successful
rob@Robs-MBP-New:/Users/rob> sudo /Applications/USBImager.app/Contents/MacOS/usbimager
2020-12-14 08:57:44.195 usbimager[950:11499] get 0x7fff80445a08
2020-12-14 08:57:54.286 usbimager[950:11499] Warning: Expected min height of view: (<NSButton: 0x7f88ea8798e0>) to be less than or equal to 30 but got a height of 32.000000. This error will be logged once per view in violation.

[bzt]

Thanks for checking out!

It looks like this is a bug in the libui library, as it calculates incorrect size for the button. However it's reported as a warning only (so it should work regardless). I'm afraid you'll have to wait a little until I can get a new machine. Once I get a MacOS build environment again, I'm planning to rewrite the interface in native ObjC to solve some other issues as well, this should be fixed with that too. One of those issues is that sometimes the window is opened in the background, and you can't see it unless you press a Cmd+Tab combo.

Cheers,
bzt

[rrobinet]

I can use my one Windows 10 laptop until you get that fixed.

In addition, it would be great if you could add an option to automatically shrink the partition size while creating an image file. I create a customized version of Raspian for a group of users by installing and configuring the standard distro on a 16+ GB microSD. While the compressed image you create is small, the expanded version mirrors the size of my 32 GB SD.
I am working through the steps to manually run gparted and dd to shrink the image before creating the img file, but it would be a very useful (and I think unique) feature if your program offered that as an automated option.

[tookys]

by rrobinet » Tue Dec 15, 2020 7:34 pm
I can use my one Windows 10 laptop until you get that fixed.

In addition, it would be great if you could add an option to automatically shrink the partition size while creating an image file. I create a customized version of Raspian for a group of users by installing and configuring the standard distro on a 16+ GB microSD. While the compressed image you create is small, the expanded version mirrors the size of my 32 GB SD.
I am working through the steps to manually run gparted and dd to shrink the image before creating the img file, but it would be a very useful (and I think unique) feature if your program offered that as an automated option.

@rrobinet I came across this project on github called pi-shrink that is able to reduce the images down to something more usable. Has worked pretty well for the custom images Ive been making.
https://github.com/Drewsif/PiShrink

You have have to run it on a linux machine is the only shortcoming i have with it.




Not sure if this is the right place to put this, but as a feature request i would like to see the imager give the option of a remote-resetable image.
Another project i came across on github is the "RaspberryPi-Factory-Reset"
https://github.com/shivasiddharth/Raspb ... tory-Reset
It creates 2 copies of a provided OS, and merges it into one image. Once booted it allows you to run a 1 line command to tell the pi to reboot itself. I've combined it with pi-shrink to help keep the images small.

This works well for me, I have pi's I keep at a remote location that I like to remote into and use. The custom image I use has ssh enabled, password changes, wifi log-in info so it reboots ready to go.


Current issues is that sometimes when re-flashing (pi4) the wifi module will stop working, but otherwise it works fine as long as its hard wired.

[bzt]

I can use my one Windows 10 laptop until you get that fixed.

Now two users reported that with build 12 the umount is working perfectly now (on High Sierra and Catalina). You should give it a try too!

In addition, it would be great if you could add an option to automatically shrink the partition size while creating an image file.

Sorry, but that's not an option. Shrinking the partition means first you have to shrink the filesystem. Adding file system drivers is a huge task and goes against the UNIX philosophy. USBImager (just like dd) does not interpret the image at all, it's only job is to read and write images to drives.

I am working through the steps to manually run gparted and dd to shrink the image

This way you're going to create a corrupted file system. You also must shrink the file system on the partition, see resize2fs (only works with ext file system family), and fatresize (only FAT file systems), ntfsresize (NTFS only) etc. There's no common tool that I know of, only proprietary solutions (like Partition Magic for example). You could try the tool @tookys recommended as well.

Here's a step by step tutorial that might be useful to you. It uses resize2fs and fdisk (note: "cat cmds.txt | fdisk" works perfectly, so it is easily scriptable).

Cheers,
bzt

[rrobinet]

I found https://github.com/Drewsif/PiShrink, a script which shrinks the partitions of a pi microSD and sets it up to resize itself upon first run.
Combined with your script, I now can generate the custom Pi microSDs I want. Thanks