How to download AND extract a .img.xz file?

[Botspot]

Yes, I know how to click a download link. I also know how to extract the image once the download is completely finished.
I also know what command to run to simultaneously download and extract a .img.ZIP file. (That's how my img downloader is the fastest in the world) For future reference here it is:

Code: Select all

wget -qO- https://downloads.raspberrypi.org/raspios_full_armhf_latest | busybox unzip - -p | dd bs=10M of=/location/for/extracted/image.img

But, I don't know how to do the same for a img.xz file. Does anyone out there know?

[DougieLawson]

Install p7zip-full and it can unzip anything.

[Botspot]

Install p7zip-full and it can unzip anything.

I'd rather not require installing anything unless I really have to. (it's going into an application for Raspbian and I don't want unnecessary dependencies)
Maybe it's possible with tar?

[DougieLawson]

It's either that or xz-utils. Pick your poison.

[jahboater]

You are using "busybox unzip - -p"
Perhaps it should be changed to "xz -d" or similar.
See "man xz" for details.

[dustnbone]

The normal command I use for .xz files is

Code: Select all

tar -xJvf file.tar.xz

The J switch is the one that specifies the compression format (LZMA2) that .xz normally uses, as opposed to the z switch that specifies gzip compression.

Anyway that tar command (or something much like it, other options may apply to your use case) in place of your unzip command should do the job.

But as Dougie said xz support might require a package not included by default. Though it's common enough that it should be part of the OS image IMHO

[Botspot]

Hmm, tar doesn't seem to like the pipe.

Code: Select all

pi@raspberrypi:~ $ wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | tar -xJvf | dd bs=10M of=/home/pi/twister.img
tar: option requires an argument -- 'f'
Try 'tar --help' or 'tar --usage' for more information.
0+0 records in
0+0 records out
0 bytes copied, 0.00760547 s, 0.0 kB/s
pi@raspberrypi:~ $ wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | tar -xJvf - | dd bs=10M of=/home/pi/twister.img
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
0+0 records in
0+0 records out
0 bytes copied, 0.699537 s, 0.0 kB/s

[Botspot]

You are using "busybox unzip - -p"
Perhaps it should be changed to "xz -d" or similar.
See "man xz" for details.

I'm familiar with xz so you think I would have tried that in a pipe! (Spent the past several days trying tar commands, all to no success)
This appears to be working:

Code: Select all

wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | xz -d - | dd bs=10M of=/home/pi/twister.img

[jahboater]

I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)

[Botspot]

I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)

In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.

[jahboater]

Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.

Cool! Sounds like you have thought about it!!!

The copy does read/write in a loop. Since we have two different devices on two different buses, Linux will do them in parallel automatically (asynchronous prefetch). Reading and writing will happen at the same time, overlapped - which gives the biggest speed in a copy. But of course writing cannot start until the first read completes and if that read is huge, time is wasted. I have seen people set the block size to the size of the file - meaning that the copy time is simply doubled!!! Aaaah!

So I don't use any special software, just let the kernel do it all for me!

[Botspot]

Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.

Sounds like you have thought about it!!!

Thank you.
Believe it or not, with enough optimization, it can be faster to download an img, extract it, and flash it to a SD card, than to flash a local, pre-extracted .img to a SD Card.

[jahboater]

Believe it or not, with enough optimization, it can be faster to download an img, extract it, and flash it to a SD card, than to flash a local, pre-extracted .img to a SD Card.

Yes I'm sure it can.

Its incredibly complicated given: different read/download speeds, decompression speeds, indeterminate overlap between reads and writes, disk write speeds, variable asynchronous disk block prefetch ranges, "sync" time, verification time, etc etc
Given all the above variables, any simple benchmark results will be meaningless ...

For example, I usually download the image to a memory disk, then uncompress it and copy it to the raw device.
So, given that the disk read times are near zero for this case only, whats the best read/write block size to give dd for the optimal overlap between reads and writes?

I gave up trying to calculate everything and produce an optimal C program!

Now I just use "cp" - its simple to use and fast enough in practice (slightly faster than dd bs=1m)

[cleverca22]

I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)

In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.

what if you dont even use dd, just `wget ... | xz -d > foo.img` ?

[Botspot]

I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)

In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.

what if you dont even use dd, just `wget ... | xz -d > foo.img` ?

I'm pretty sure that a simple '>' would do it one bit at a time. Even if it didn't, without a buffer the whole pipe is sensitive to slowdowns.
For example, when flashing directly from the internet, I found that some USB devices occasionally did a flush or something, and that causes a periodic slowdown. With two consecutive buffers, the pipe could keep downloading and extracting for a while in the background.
But without a buffer, the whole pipe would immediately slow down at once, and would rarely ever reach full speed again.

[jahboater]

I'm pretty sure that a simple '>' would do it one bit at a time.

Its writes with a block size of 8KB (probably BUFSIZ). But likely the speed is limited by the decompress time, as the writes will go straight into memory (the disk cache). Sync must be used afterwards.

You can see the I/O reads and writes for any command with strace, for example:

Code: Select all

strace xz -d Python-3.9.0.tar.xz - >tmp

But you cant see whats going on behind the scenes in the kernel.

Note that cp optimizes prefetch (with POSIX_FADV_SEQUENTIAL ) but dd does not.
Prefetch is key to getting fast overall copy times because it allows the writes to proceed at the same time as the reads, hopefully nearly halving the total time.
That's one reason why commands like cp use a smallish block size (128 KB) because writing cannot commence until the first block read has completed. Giving a huge block size to dd removes much, or all, of the overlap between the reads and the writes!