bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

USB image writer

Sun Feb 16, 2020 9:37 am

Hi,

First of all, if there's a better forum for supporting applications than "General discussion", then please feel free to move this topic there.

I'm a very advanced user and I'm very happy with the command line. I can write compressed images on-the-fly with pipes and dd. However I have some non-cli-wizzard friends who wants to use RPis too, and frankly I'm bored of creating SD cards for them. The officially recommended image writer tool is ridiculously bloated (hundreds of megabytes, dear Lord!), has advertisements and now it raises privacy concerns as well (seriously?). Long story short, I've decided to create a small and useful tool which is multi-platform and follows the K.I.S.S. principle.

So let me introduce USBImager to write compressed disk images to USB drives.

Features
  • Non-bloated, small. When I say small, I mean really-really super small, about 100K on each platform.
  • Portable executable, dependency free and no installation required
  • Free and Open Source, MIT licensed
  • Can read raw images and .gz, .bz2, .xz (LZMA), .zip (PKZIP and ZIP64) compressed images
  • Tries to be user-friendly by disallowing to write to the system disk
  • Can verify what's written by comparing the disk to the image
  • Minimalist, yet very informative interface
  • Estimates remaining time
Screenshots
Image

Please note that I don't get paid for this, I only did it in my spare time. I really wish with some help we can make this tool suitable to be the official USB image writer for the Raspberry Pi. But I cannot guarantee anything, and under no circumstances can I held responsible for potentially destroyed filesystems and loss of data using USBImager.

Right now this tool is experimental. It compiles without errors on all platforms, and I did all the basic tests. I'm now looking for brave volunteers for beta testers and contributors. I have only limited set of USB sticks, and my access to MacOSX and Windows machines is also limited, so testing the tool on those platforms is extremely problematic for me, any help would be appreciated.

As far as I know, drives are detected fine on all platforms, including disk capacity, vendor and model strings. I've noticed that USB drives are not always marked as removable under Windows, and I've found some forums where people are saying the same for Linux (I haven't run into this, but let me know). Unmounting and locking (claiming) should also work fine, however I only now this for sure under Linux. I've noticed phtread issues under MacOSX.

For hacking the source, it is extremely simple, no more than a handful of files, and written in ANSI C. The input.c file contains the code to read and uncompress the input file. The output is handled in disks_*.c, one for each platform. Finally the user interface is managed in the main_*.c files. Data decompression is tested pretty well, but if you want to play with this, you can edit the Makefile and set DISKS_TEST to 1. This will add a test.bin file "USB device" target. Running usbimager with `-v` command line argument, it will output verbose messages.

Cheers,
bzt

mattmiller
Posts: 2215
Joined: Thu Feb 05, 2015 11:25 pm

Re: USB image writer

Sun Feb 16, 2020 12:27 pm

Failed on my Win10 trying to write latest full raspbian to a brand new 64GB card

Got popup dialog saying "an error occurred when writing to the target device" when I pressed the write button

User avatar
B.Goode
Posts: 9840
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: USB image writer

Sun Feb 16, 2020 1:35 pm

bzt wrote:
Sun Feb 16, 2020 9:37 am
The officially recommended image writer tool is ridiculously bloated (hundreds of megabytes, dear Lord!), has advertisements and now it raises privacy concerns as well (seriously?).

[ ... ]

Please note that I don't get paid for this



There is always room for another user-supported application. Thank for sharing what you are attempting.

But does balenaEtcher (presumably the alternative you are critiquing) have 'advertisments', or does it have some self-promotion flash screens that appear while the image is being written? (In a very similar fashion to the Raspberry Pi NOOBS Installer.)

It isn't possible to comment on whether to take privacy concerns seriously without a reference to what it is that concerns you.

And I've never been asked to pay to download or use balenaEtcher - like you, Balena appear to have decided to create a tool for their own needs and make it widely available.

MisterEd
Posts: 137
Joined: Mon Apr 16, 2018 5:28 am
Location: Huntsville, AL USA

Re: USB image writer

Sun Feb 16, 2020 1:53 pm

I tried the same thing on my Windows 10 laptop with uSD Card: SanDisk Ultra (32GB)

From: C:\Users\me\Downloads\2020-02-13-raspbian-buster-full.zip
To: D: [30.7 GiB] Mass Storage Device

I get the error: "An error occurred while writing to the target device."

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Sun Feb 16, 2020 7:34 pm

Hi,
mattmiller wrote:Failed on my Win10 trying to write latest full raspbian to a brand new 64GB card
MisterEd wrote:I get the error: "An error occurred while writing to the target device."
As I wrote, the Windows port is (and so is the MacOSX port) experimental right now. I've implemented the DeviceIoControl calls according to the Miscrosoft documentation, but it's not perfect yet. Right now I'm comparing my source to the win32 disk image writer's source to spot the difference, and figure out what are the right arguments for the locking, dismounting. Stay tuned!

B.Goode wrote:
Sun Feb 16, 2020 1:35 pm
There is always room for another user-supported application. Thank for sharing what you are attempting.
Thanks, but I'm not only attempting, it already works perfectly under Linux, and I am certain that the Windows version will be production ready pretty soon. The MacOSX version is a bit trickier, I left that last.
B.Goode wrote:
Sun Feb 16, 2020 1:35 pm
But does balenaEtcher (presumably the alternative you are critiquing) have 'advertisments', or does it have some self-promotion flash screens that appear while the image is being written? (In a very similar fashion to the Raspberry Pi NOOBS Installer.)

It isn't possible to comment on whether to take privacy concerns seriously without a reference to what it is that concerns you.
The concerns were raised by its users, not by me:
Etcher 1.4.4 Ignores Privacy Setting
Serious Privacy Concerns with Etcher 1.4.4
and also Raspberry Pi community members right on this very forum has concerns:
What is a good replacement for Etcher (ads and ignores privacy)
and we have this in the official Raspberry Pi issue which suggests in the EU it is illegal what etcher does:
balenaEtcher and privacy/GDPR

If you ask me, if a disk writer tool is doing any network connections at all, then it's spying on you. My tool does not use net, because internet is NOT NEEDED for writing disk images in the first place, period. K.I.S.S. you know.

And if it weren't for these issues and the telemetry, I still wouldn't never ever download such a bloated, hundreds of megabytes tool just to write out an image file. ;-) That's just sick and insane, it's bigger than most of the OS images I use!
B.Goode wrote:
Sun Feb 16, 2020 1:35 pm
And I've never been asked to pay to download or use balenaEtcher - like you, Balena appear to have decided to create a tool for their own needs and make it widely available.
Oh, but you pay for using etcher plenty. Not with money, but with your data. You also loose your privacy, which is way too high price for such a simple tool.

Cheers,
bzt

User avatar
scruss
Posts: 3072
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: USB image writer

Sun Feb 16, 2020 8:47 pm

bzt wrote:
Sun Feb 16, 2020 7:34 pm
Thanks, but I'm not only attempting, it already works perfectly under Linux, …
Not so much, I'm afraid: I got this running under Ubuntu 19.10 x86_64:
Screenshot from 2020-02-16 15-38-27.png
An error occurred … Ubuntu 19.10
Screenshot from 2020-02-16 15-38-27.png (14.55 KiB) Viewed 2930 times
Update: it seems to need sudo under Ubuntu, though nothing about this in the docs. It should run as user then prompt for password when it needs elevated permissions.

Also it's misreporting media size. I put up an issue on your gitlab.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

pica200
Posts: 216
Joined: Tue Aug 06, 2019 10:27 am

Re: USB image writer

Sun Feb 16, 2020 10:22 pm

Something that would make your image writer stand out of all the others would be proper fake testing using the code from f3. Maybe not by default but as an option.

See this and my post below it:
viewtopic.php?f=63&t=252758&p=1612817#p1612782

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Mon Feb 17, 2020 4:08 am

Thank you guys for checking out!

I've fixed access under Windows, it should be ok now. I've also added more sophisticated error messages, including the ones reported by the system (those might be translated). I've also workarounded the pthread issue under MacOSX, unfortunately the progress bar is not updating instantly as it should be, but the tool is working, which is more important. Unmounting works perfectly on both MacOSX and Linux, and lock and dismount on Windows uses the same DeviceIoControl arguments as win32 disk imager. If win32 disk imager can write an USB device, then USBImager will too.

@mattmiller, @MisterEd: could you please give it another spin? I don't have Win10, I can't test it myself on that OS. Thanks!
scruss wrote:Update: it seems to need sudo under Ubuntu, though nothing about this in the docs.
Nope, no need for sudo, just write permission on the target device obviously. I've added more error messages, now you'll probably see "Permission denied" instead of "Error". Not sure what should I put into the docs, you can find this in the Ubuntu docs, under basic file permissions. Setting up user privileges to access files is part of system management, not the tool's job (for example you could add your user to the "disk" group, and then you'll have write access to all disks). The command line utility "dd" won't work either if you can't write the device file.
scruss wrote:It should run as user then prompt for password when it needs elevated permissions.
It does not need elevated permissions. Requiring write permission on a device file is conceptually different than running the entire application as root. However it could prompt for password and run as root if opening the device for writing fails with "Permission denied" error. I'll look into that.
scruss wrote:Also it's misreporting media size. I put up an issue on your gitlab.
Thanks, it's fixed. Foolish rounding issue made me forget that Linux reports size in 512 byte blocks.
pica200 wrote:Something that would make your image writer stand out of all the others
What others? I couldn't find any simple multi-platform disk imager capable of writing images from compressed files, that's why I wrote this. Believe me, I'm lazier than that, I wouldn't start this project if I could find viable alternatives googling around a day or two.
pica200 wrote:would be proper fake testing using the code from f3. Maybe not by default but as an option.

See this and my post below it:
viewtopic.php?f=63&t=252758&p=1612817#p1612782
That project has nothing to do with imaging. It is a flash device tester, while USBImager is an image writer. Use different tools for different purposes, that's the UNIX philosophy. You don't want to hammer a screw or use a screwdriver on a nail, do you? :-)

Cheers,
bzt

MisterEd
Posts: 137
Joined: Mon Apr 16, 2018 5:28 am
Location: Huntsville, AL USA

Re: USB image writer

Mon Feb 17, 2020 6:07 am

I just downloaded the latest version. This time right after I select the Write button I get "Write verification failed".

I reformatted the card and successfully wrote the image with Etcher.

BTW, would it be possible to put version numbers in the usbimager.exe file so that we know which version we are using. Right now both downloads show version 1.0.0.0. Currently the only way for me to to tell the difference is the file size.

User avatar
rpdom
Posts: 16746
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: USB image writer

Mon Feb 17, 2020 7:02 am

Would an option to write the image direct from a known download site be an idea? For example, have an optional menu that lists the three current Raspbian options (Lite, Full and Full with Recommended Software) and will then go to the Raspberry Pi download section and pull the file down and write it directly to the card.
Unreadable squiggle

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Mon Feb 17, 2020 9:35 am

MisterEd wrote:
Mon Feb 17, 2020 6:07 am
I just downloaded the latest version. This time right after I select the Write button I get "Write verification failed".
And does it work without the verification checked? If so then I know what the problem is.
MisterEd wrote:I reformatted the card and successfully wrote the image with Etcher.
Yes, I understand that etcher works, so does its spyware functionality :-P Keep in mind that etcher is a software with history and has a stable release, while this one is only a one day old project with just an experimental beta version for now.
MisterEd wrote:BTW, would it be possible to put version numbers in the usbimager.exe file so that we know which version we are using. Right now both downloads show version 1.0.0.0. Currently the only way for me to to tell the difference is the file size.
This is a rolling release in beta stage, subject to rapid development and bugfixes. I'll handle the versioning accurately as soon as the first stable comes out. Until then use the commit number or the file's date, please. But I'll change the version to 0.0.0.1, and only use 1.0.0.0 for the first stable release, thanks for pointing out.
rpdom wrote:Would an option to write the image direct from a known download site be an idea?
Not a chance. I hate huge bloated software that tries to do everything. That's not the UNIX way. Do you remember WinAmp and Nero Burning Room for example? At the beginning, their first versions did one thing only, and they were awesome. Everybody liked them and everybody used them. Then they began to include more and more features, became larger and slower, up to the point where nobody ever wanted to use them any more.

Besides, I prefer to use torrent over http to download images, and I also use many OSes, not just raspbian. The problem with your approach is (this is an existing problem to all similar solutions):
1. who would decide which URLs to include?
2. who would keep the list updated? Add new URLs, remove dead ones, update version in URLs to the latest etc.
3. who would decide the order? What if someone prefers Haribote or Gentoo? Someone's feelings will get hurt, no matter what you do. Just take a look at the websearch engine choosers. Even the big players with huge human resources and lots of money can't solve this simple issue.
4. writing images directly from socket streams is risky at best. What if the channel gets a tcp reset at 99%? You would have to download the entire image again, wasting your own time and the server's resources as well.
5. there are simple tools (wget, curl, right click in any browser, and also trillions of download managers etc.) that can already do the job.

FYI, you can use

Code: Select all

wget https://(some neat OS image of your liking) -O - | /usr/bin/usbimager
and select "/dev/stdin" in USBImager as source if you really want to write images without saving them locally.

Further reading on the topic: UNIX Philosophy:
This is the Unix philosophy: Write programs that do one thing and do it well.
...
3. Rule of Composition: Design programs to be connected to other programs.
...
5. Rule of Simplicity: Design for simplicity; add complexity only where you must.
6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
...
So I rather keep this tool to do one job, and do that well, and let the user get the image, whatever they want, and however they want.

Cheers,
bzt

User avatar
rpdom
Posts: 16746
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: USB image writer

Mon Feb 17, 2020 10:26 am

bzt wrote:
Mon Feb 17, 2020 9:35 am
rpdom wrote: Would an option to write the image direct from a known download site be an idea?
Not a chance. I hate huge bloated software that tries to do everything. That's not the UNIX way. Do you remember WinAmp and Nero Burning Room for example? At the beginning, their first versions did one thing only, and they were awesome. Everybody liked them and everybody used them. Then they began to include more and more features, became larger and slower, up to the point where nobody ever wanted to use them any more.
Fair enough. No, I don't remember WinAmp or Nero Burning Room. I never used them. I dropped Windows decades ago.

I'll stick to my method of downloading and writing then. I have one bash script that downloads the current Raspbian images (I have no interest in any other OS currently), and recompresses them with xz to save space. I have another script that gives me the option to choose any of those images (going back a few years) and will write them to a card, set up some local options (wifi/hostname/apt preferences etc.) and expand the filesystem in advance so the first boot will be quicker.
Unreadable squiggle

tvjon
Posts: 765
Joined: Mon Jan 07, 2013 9:11 am

Re: USB image writer

Mon Feb 17, 2020 11:22 am

bzt wrote:
Sun Feb 16, 2020 9:37 am
...
Right now this tool is experimental. It compiles without errors on all platforms, and I did all the basic tests. I'm now looking for brave volunteers for beta testers and contributors. I have only limited set of USB sticks, and my access to MacOSX and Windows machines is also limited, so testing the tool on those platforms is extremely problematic for me, any help would be appreciated.

...
Cheers,
bzt
Yesterday morning I built usbimager for RPi Raspbian Buster, as that's the only platform I would ever use it on, & of course I couldn't use your supplied binaries, as they assume X86, but no time yesterday to post feedback.

It wouldn't read a source file from a NTFS volume, but I see the new version corrects that, perhaps permissions checking, I could always read & write fine to that drive.

So, with the new version, it started up fine (pic' attached), & I managed to copy a couple of textfiles as tests, so in fact a glorified linux

cp existing_file_from_my_drive new_copy_to_your_test-disc

Next, after the third invocation of the program, interestingly the new version does exactly as the original did; it seg-faults.

I made a work-around to that on the original by inserting a

sleep(1);

prior to the

uiMain();

call.

It can be seen from the warnings (I'm posting the originals from yesterday as the new version is much the same) that gcc is unhappy about the "lu"'s types, but unlike you, I'm not so advanced as to determine if that's a red herring or not.

Sadly, the sleep doesn't allow the new version to run any longer, & today I have no time to investigate further, but hopefully the feedback will be useful?


(gdb) run
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[Detaching after fork from child process 9250]
[Detaching after fork from child process 9251]
[Detaching after fork from child process 9252]
[New Thread 0xb5366210 (LWP 9253)]
[New Thread 0xb49ff210 (LWP 9254)]

Thread 1 "usbimager" received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/arm/armv6/strlen.S:26
26 ../sysdeps/arm/armv6/strlen.S: No such file or directory.
(gdb)


During make:

main_unix.c: In function 'writerRoutine':
main_unix.c:95:109: warning: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'size_t' {aka 'unsigned int'} [-Wformat=]
bose) printf("writerRoutine() numberOfBytesRead %d numberOfBytesWritten %lu\n",


main_unix.c:101:81: warning: format '%lu' expects argument of type 'long unsigned int', but argument 2 has type 'size_t' {aka 'unsigned int'} [-Wformat=]
if(verbose) printf(" numberOfBytesVerify %lu\n", numberOfBytesVerify);


disks_linux.c: In function 'disks_refreshlist':
disks_linux.c:103:53: warning: format '%ld' expects argument of type 'long int', but argument 5 has type 'uint64_t' {aka 'long long unsigned int'} [-Wformat=]
snprintf(str, sizeof(str)-1, "%s [%ld.%ld GiB] %s %s", de->d_name,

disks_linux.c:103:57: warning: format '%ld' expects argument of type 'long int', but argument 6 has type 'uint64_t' {aka 'long long unsigned int'} [-Wformat=]
snprintf(str, sizeof(str)-1, "%s [%ld.%ld GiB] %s %s", de->d_name,

You weren't kidding about the libraries; attached is the huge list of them.

Anyway, thank you for providing the source. It's a useful example referencing meson & ninja to build libui, & libui looks very comprehensive & may be useful for other applications on RPi in the future.
Attachments
libraries.zip
(1.14 KiB) Downloaded 25 times
usbimager-newver.png
usbimager-newver.png (17.87 KiB) Viewed 2744 times
original-won't-read-from-ntfs-volume.png
original-won't-read-from-ntfs-volume.png (19.9 KiB) Viewed 2744 times

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Mon Feb 17, 2020 1:24 pm

Hi,

@rpdom: WinAmp was the de facto standard MP3 player on desktops (XMMS and now Audacious mimics it's look'n'feel). At first it was a small tool, lately come with a webbrowser integrated into it. Don't ask me who have thought that having a browser inside a music player app is a great idea...

Yeah, that's a good workflow you have there! I've added XZ decompressor to USBImager because I also do like to recompress images for smaller sizes :-) And of the three compression filters, XZ is not only the most compact, but that's the fastest to extract too, by a lot.

@MisterEd: write verification should work under Win10 now. If not, then please start "usbimager.exe -v" from cmd.exe, and paste the Debug Console's output here.

@scruss: asking for password to elevate permissons won't work. The GTK library does not allow it. I've added description to the README.md that you should add your user to the "disk" group (or "operator" group under MacOSX), and use "Run As Administrator" under Windows if you experience permission denied errors.

@tvjon: thanks for checking out! Please download the latest, many of these problems have already been fixed.

This code should compile and work on ARM, that's for sure. I must fix this!

1. about arbitrary files: Linux demands that the write must be multiple of 512 bytes for block devices. USBImager now adds zero padding to fulfill this requirement. This eliminates many error messages.
2. the "lu"s has already been fixed, because I had the same warnings under MacOSX
3. about the SIGSEGV in strlen, that's very interesting, could you please recompile with "-g" in CFLAGS and do a backtrace in gdb? This is a critical issue
4. the new version provides lot more sophisticated error messages than those you have on the screenshots.

Also, you can run usbwriter with the "-v" verbose flag, that will print lots of useful information to stdout (exact errno values among others). I finally managed to get this working under Windows too, it now opens a second window by using AllocConsole() and redirects printf there.

Yes, libui seemed promising, but I'm thinking of removing it from the equation entirely:

1. I couldn't statically link with it under Windows (not even its own examples), because the win32 part was written in C++ instead of C, and has many DLL dependencies and special ABI quirks. So I had to use a native win32 app from the start.

2. I've noticed that the MacOSX port has issues with pthread. Right now I've removed that, which means the progress bar does not update as it should (I call [NSApp updateWindows], and I also do a CFRunLoopRun() to make sure the events are dispatched. But it is not looking good, maybe it has something to do with the event timestamps, as events are processed in bulks). I think I'll use a native Cocoa app here too, that would solve the lagging progress bar.

3. The Linux port of libui uses GTK, which has many dependencies as you've noticed. Thankfully this usually works, there's no problem with GTK applications on mainline distros. However I already have a native app for Windows, and if I write a native app for Mac, then there would be no reason to use a portability layer for Linux only. I could write a native app for Linux too, using only Xlib (or Wayland). It wouldn't look so great as the GTK version, but it would only depend on just one library.

Cheers,
bzt

MisterEd
Posts: 137
Joined: Mon Apr 16, 2018 5:28 am
Location: Huntsville, AL USA

Re: USB image writer

Mon Feb 17, 2020 2:14 pm

Here are the three versions I have tried for usbimager.exe:
1. 2/16/2020 8:15 AM 164,878
2. 2/17/2020 4:08 AM 166,926
3. 2/17/2020 11:34 AM 166,926

You are right about why version #2 failed. Version 3 seems to work OK.

Version #2:
- without verification: ran successfully to completion
- with verification: failed immediately with "Write verification failed".

Version #3
- without verification: not tested
- with verification: ran successfully to completion

mattmiller
Posts: 2215
Joined: Thu Feb 05, 2015 11:25 pm

Re: USB image writer

Mon Feb 17, 2020 2:30 pm

(started this post at 10am - got distracted)
re-downloaded and tried again - got "write verification failed" when I pressed write button

deselected the verify option, pressed write and it seems to have started writing to disc
...
and it worked (tested using it on a Pi)
(will try again with current version)

mattmiller
Posts: 2215
Joined: Thu Feb 05, 2015 11:25 pm

Re: USB image writer

Mon Feb 17, 2020 2:48 pm

Version #3
- without verification: not tested
- with verification: ran successfully to completion
Did same test - same result for me as well

User avatar
scruss
Posts: 3072
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: USB image writer

Mon Feb 17, 2020 5:29 pm

bzt wrote:
Mon Feb 17, 2020 1:24 pm
@scruss: asking for password to elevate permissons won't work. The GTK library does not allow it. I've added description to the README.md that you should add your user to the "disk" group (or "operator" group under MacOSX), and use "Run As Administrator" under Windows if you experience permission denied errors.
Adding a user to the disks group is effectively the same as running everything as root. Debian doesn't enable that by default for safety reasons. I wouldn't do it personally, and I wouldn't recommend anyone do it either. As a member of disks, one slip of the keyboard and you've hosed your OS.

Etcher and Gnome Disks manage the permissions escalation gracefully, so it can be done. Yes, I know your project is very new, and it's remarkably great already, but it won't replace Etcher until it does that. When I'm doing Intro to Raspberry Pi sessions to the general public, the OS imaging has to be something you can explain in one simple sentence or you'll lose people or make them afraid to try.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

pica200
Posts: 216
Joined: Tue Aug 06, 2019 10:27 am

Re: USB image writer

Mon Feb 17, 2020 8:16 pm

bzt wrote:
Mon Feb 17, 2020 4:08 am
pica200 wrote:would be proper fake testing using the code from f3. Maybe not by default but as an option.

See this and my post below it:
viewtopic.php?f=63&t=252758&p=1612817#p1612782
That project has nothing to do with imaging. It is a flash device tester, while USBImager is an image writer. Use different tools for different purposes, that's the UNIX philosophy. You don't want to hammer a screw or use a screwdriver on a nail, do you? :-)
I think you missed the point. Even if it only included the quick test it would avoid lots of support due to fake cards. In the end it's about easy usage and safety. Explaining newcomers the usage of a basic tool and enforcing checks is easier than having them deal with 2 separate tools.

The UNIX philosophy makes sense in lots of cases but i think here it's better to have an all-in-one. Besides that your tool already does 2 things: uncompressing and image writing :p

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Mon Feb 17, 2020 9:35 pm

Hi,

@MisterEd, @mattmiller:
ran successfully to completion
Thank you guys! Let me know if you run into something else!




scruss wrote:
Mon Feb 17, 2020 5:29 pm
Adding a user to the disks group is effectively the same as running everything as root.
Auch. I'm a security expert for more than a decade now, so I'm going to pretend you never wrote this.
scruss wrote:Debian doesn't enable that by default for safety reasons. I wouldn't do it personally, and I wouldn't recommend anyone do it either. As a member of disks, one slip of the keyboard and you've hosed your OS.
Your reasoning is invalid. Adding your user to the disk group does not risk security, and no "slip of the keyboard" can cause trouble. You'll have write permission to a handful of files more (all located under /dev), that's all. Running everything as root on the other hand does involve a HUGE security risk, and you could really bust your OS with a slip on the keyboard. Running as root is considered to be a really bad practice (not just by me, by the whole industry, the GTK developers included).
scruss wrote:Etcher and Gnome Disks manage the permissions escalation gracefully, so it can be done.
Nope, it cannot be done. USBImager uses libui, which in turn realizes the UI using GTK under Linux. Try:

Code: Select all

$ chgrp disk usbimager 
$ chmod g+s usbimager 
$ ./usbimager 
(process:68604): Gtk-WARNING **: 21:50:28.983: This process is currently running setuid or setgid.
This is not a supported use of GTK+. You must create a helper
program instead. For further details, see:

    http://www.gtk.org/setuid.html

Refusing to initialize GTK+.
$
(Calling this a "WARNING" when GTK completely refuses to run is another issue, but let's not dive into that.) Etcher does not use GTK (it's a web based app), and gnome workarounds this by using a daemon and a separate interface application talking through pipes. Both solutions are impossible for a portable, single file executable. I'd have to rewrite USBImager using a different UI framework than GTK, but still there would be issues with that, for example Fedora deletes all executables with setuid root without asking.

FYI, to provide what you ask, an app must be installed with setuid bit set. But if you have that, then you can have a disk setgid too instead which would allow write access to block devices, so it won't get permission denied errors in the first place.
scruss wrote:Yes, I know your project is very new, and it's remarkably great already, but it won't replace Etcher until it does that. When I'm doing Intro to Raspberry Pi sessions to the general public, the OS imaging has to be something you can explain in one simple sentence or you'll lose people or make them afraid to try.
Thanks! You are right, however I don't see the point in your reasoning: the general public uses Windows, where this permission denied issue doesn't exists, and more advanced users who dare to run debian are well aware about UNIX file permissions. Worst case scenario you can advise newbies in Intro to Raspberry Pi to run "sudo usbimager" (wouldn't be the first nor the only command with sudo), however I would afraid to try that lot more than adding my user to the "disk" group (but I'm a security expert, not a common, average linux user).





pica200 wrote:I think you missed the point. Even if it only included the quick test it would avoid lots of support due to fake cards. In the end it's about easy usage and safety. Explaining newcomers the usage of a basic tool and enforcing checks is easier than having them deal with 2 separate tools.
You're absolutely right. I don't question that, I just don't understand reading back what's written and comparing it to the original buffer why doesn't suffice as a quick test for you. It does notify the user about fake cards, so I don't get it.

That being said, I'm not against using a more robust testing when "verify" checkbox is checked. You can convince me to add more advanced test, but I definitely don't want to complicate the user interface. Having a single checkbox is about "easy usage".
pica200 wrote:The UNIX philosophy makes sense in lots of cases but i think here it's better to have an all-in-one.
But it is already an all-in-one. You can check the verify checkbox to have safety (we can talk about having more tests performed).
pica200 wrote:Besides that your tool already does 2 things: uncompressing and image writing :p
No, my tool does ONE thing: writing out a local file to a disk. If that local file happens to be in a specific format (stream compressed), then it must be interpreted accordingly. Same as with "zcat" for example.

Cheers,
bzt

User avatar
scruss
Posts: 3072
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: USB image writer

Tue Feb 18, 2020 10:19 pm

bzt wrote:
Mon Feb 17, 2020 9:35 pm
scruss wrote:
Mon Feb 17, 2020 5:29 pm
Adding a user to the disks group is effectively the same as running everything as root.
Auch. I'm a security expert for more than a decade now, so I'm going to pretend you never wrote this.
You don't have to imagine I wrote it: I paraphrased it from the Debian docs:
· disk: Raw access to disks. Mostly equivalent to root access.


Security implications

The group disk can be very dangerous, since hard drives in /dev/sd* and /dev/hd* can be read and written bypassing any file system and any partition, allowing a normal user to disclose, alter and destroy both the partitions and the data of such drives without root privileges. Users should never belong to this group.
So, no — it's not reasonable to require users to be a member of disk.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

GlowInTheDark
Posts: 446
Joined: Sat Nov 09, 2019 12:14 pm

Re: USB image writer

Wed Feb 19, 2020 12:29 am

So, no — it's not reasonable to require users to be a member of disk.
How do other, similar programs (e.g., the frequently mentioned on the forum "Etcher" program) do this?

What do they require of users in order to be able to write to the raw disk devices (under Unix and Unix-like OSes) ?
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

User avatar
scruss
Posts: 3072
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: USB image writer

Wed Feb 19, 2020 2:04 am

They use something like SGID or gksu to elevate permissions temporarily.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

GlowInTheDark
Posts: 446
Joined: Sat Nov 09, 2019 12:14 pm

Re: USB image writer

Wed Feb 19, 2020 10:06 am

scruss wrote:
Wed Feb 19, 2020 2:04 am
They use something like SGID or gksu to elevate permissions temporarily.
OK. IIRC, there was some reason why OP didn't want to go that route for his program.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

bzt
Posts: 542
Joined: Sat Oct 14, 2017 9:57 pm

Re: USB image writer

Wed Feb 19, 2020 10:10 am

scruss wrote:
Tue Feb 18, 2020 10:19 pm
So, no — it's not reasonable to require users to be a member of disk.
ROTFL! It is not reasonable to expect to write to a file without write permission...
disk: Raw access to disks. Mostly equivalent to root access.
If you really quoted that from the Debian docs, then the Debian docs are wrong. Also note the mostly in the sentence. Just think about it: being a member of the disk group gives you raw access to disks, but nothing else. Running with UID 0 (that's what sudo and gksu does) bypasses not just the file access permissions for the disk devices, but it also does bypass every other security checks as well!!! For example being a member of disk does not allow you to list files under "/var/log/private" or to modify "/etc/shadow". If you are root, you can do both without the slightest issue. And we haven't talked about other, non-filesystem-related system calls that only root allowed to use...
scruss wrote:They use something like SGID or gksu to elevate permissions temporarily.
Don't make fool of yourself more, please. Have you actually tried that? You can't use SGID with GTK, and I've already told you that. But here it goes again: https://www.gtk.org/setuid.html.
GlowInTheDark wrote:What do they require of users in order to be able to write to the raw disk devices (under Unix and Unix-like OSes) ?
Write permission to the disk is among them, that's for sure.
GlowInTheDark wrote:OK. IIRC, there was some reason why OP didn't want to go that route for his program.
No. I'm perfectly fine with that, I think it's much better solution than sudo, it is the GTK framework that doesn't allow setgid.

@tvjon: I've checked, and there's no way my code called strlen() with a NULL argument. Must have been inside libui somewhere.

Cheers,
bzt

Return to “General discussion”