User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:14 pm

Yes, I know how to click a download link. I also know how to extract the image once the download is completely finished.
I also know what command to run to simultaneously download and extract a .img.ZIP file. (That's how my img downloader is the fastest in the world) For future reference here it is:

Code: Select all

wget -qO- https://downloads.raspberrypi.org/raspios_full_armhf_latest | busybox unzip - -p | dd bs=10M of=/location/for/extracted/image.img
But, I don't know how to do the same for a img.xz file. Does anyone out there know?
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
DougieLawson
Posts: 40516
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:20 pm

Install p7zip-full and it can unzip anything.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:24 pm

DougieLawson wrote:
Mon Oct 12, 2020 9:20 pm
Install p7zip-full and it can unzip anything.
I'd rather not require installing anything unless I really have to. (it's going into an application for Raspbian and I don't want unnecessary dependencies)
Maybe it's possible with tar?
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
DougieLawson
Posts: 40516
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:27 pm

It's either that or xz-utils. Pick your poison.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

User avatar
jahboater
Posts: 6511
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:29 pm

You are using "busybox unzip - -p"
Perhaps it should be changed to "xz -d" or similar.
See "man xz" for details.
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

dustnbone
Posts: 428
Joined: Tue Nov 05, 2019 2:49 am

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:29 pm

The normal command I use for .xz files is

Code: Select all

tar -xJvf file.tar.xz
The J switch is the one that specifies the compression format (LZMA2) that .xz normally uses, as opposed to the z switch that specifies gzip compression.

Anyway that tar command (or something much like it, other options may apply to your use case) in place of your unzip command should do the job.

But as Dougie said xz support might require a package not included by default. Though it's common enough that it should be part of the OS image IMHO

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:33 pm

Hmm, tar doesn't seem to like the pipe.

Code: Select all

pi@raspberrypi:~ $ wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | tar -xJvf | dd bs=10M of=/home/pi/twister.img
tar: option requires an argument -- 'f'
Try 'tar --help' or 'tar --usage' for more information.
0+0 records in
0+0 records out
0 bytes copied, 0.00760547 s, 0.0 kB/s
pi@raspberrypi:~ $ wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | tar -xJvf - | dd bs=10M of=/home/pi/twister.img
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
0+0 records in
0+0 records out
0 bytes copied, 0.699537 s, 0.0 kB/s
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:35 pm

jahboater wrote:
Mon Oct 12, 2020 9:29 pm
You are using "busybox unzip - -p"
Perhaps it should be changed to "xz -d" or similar.
See "man xz" for details.
I'm familiar with xz so you think I would have tried that in a pipe! (Spent the past several days trying tar commands, all to no success)
This appears to be working:

Code: Select all

wget -qO- https://archive.org/download/twister-osv-1-8-4.img/TwisterOSv1-8-4.img.xz | xz -d - | dd bs=10M of=/home/pi/twister.img
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
jahboater
Posts: 6511
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:40 pm

I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 9:45 pm

jahboater wrote:
Mon Oct 12, 2020 9:40 pm
I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)
In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
jahboater
Posts: 6511
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 10:01 pm

Botspot wrote:
Mon Oct 12, 2020 9:45 pm
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.
Cool! Sounds like you have thought about it!!!

The copy does read/write in a loop. Since we have two different devices on two different buses, Linux will do them in parallel automatically (asynchronous prefetch). Reading and writing will happen at the same time, overlapped - which gives the biggest speed in a copy. But of course writing cannot start until the first read completes and if that read is huge, time is wasted. I have seen people set the block size to the size of the file - meaning that the copy time is simply doubled!!! Aaaah!

So I don't use any special software, just let the kernel do it all for me!
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 10:08 pm

jahboater wrote:
Mon Oct 12, 2020 10:01 pm
Botspot wrote:
Mon Oct 12, 2020 9:45 pm
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.
Sounds like you have thought about it!!!
Thank you.
Believe it or not, with enough optimization, it can be faster to download an img, extract it, and flash it to a SD card, than to flash a local, pre-extracted .img to a SD Card.
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
jahboater
Posts: 6511
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: How to download AND extract a .img.xz file?

Mon Oct 12, 2020 10:31 pm

Botspot wrote:
Mon Oct 12, 2020 10:08 pm
Believe it or not, with enough optimization, it can be faster to download an img, extract it, and flash it to a SD card, than to flash a local, pre-extracted .img to a SD Card.
Yes I'm sure it can.

Its incredibly complicated given: different read/download speeds, decompression speeds, indeterminate overlap between reads and writes, disk write speeds, variable asynchronous disk block prefetch ranges, "sync" time, verification time, etc etc
Given all the above variables, any simple benchmark results will be meaningless ...

For example, I usually download the image to a memory disk, then uncompress it and copy it to the raw device.
So, given that the disk read times are near zero for this case only, whats the best read/write block size to give dd for the optimal overlap between reads and writes?

I gave up trying to calculate everything and produce an optimal C program!

Now I just use "cp" - its simple to use and fast enough in practice (slightly faster than dd bs=1m)
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

cleverca22
Posts: 2418
Joined: Sat Aug 18, 2012 2:33 pm

Re: How to download AND extract a .img.xz file?

Tue Oct 13, 2020 4:12 am

Botspot wrote:
Mon Oct 12, 2020 9:45 pm
jahboater wrote:
Mon Oct 12, 2020 9:40 pm
I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)
In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.
what if you dont even use dd, just `wget ... | xz -d > foo.img` ?

User avatar
Botspot
Posts: 1635
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas! :)
Contact: Website

Re: How to download AND extract a .img.xz file?

Tue Oct 13, 2020 12:45 pm

cleverca22 wrote:
Tue Oct 13, 2020 4:12 am
Botspot wrote:
Mon Oct 12, 2020 9:45 pm
jahboater wrote:
Mon Oct 12, 2020 9:40 pm
I'm glad "xz -d" works!
Just for my interest, why are you using "bs=10M" for the dd command?
The cp command (which is very fast in my tests) uses just 128KB block size (see strace cp ....)
In my tests, I found it gives a bit of a buffer. It usually cut down on download time by around 3 seconds. (and when you want to have the world's fastest img downloader, every second counts.)
Additionally, in my script I go a couple steps further by loading the first 100MB of the img into cache first, and also it uses the buffer package to overcome I/O bottlenecks for maximum efficiency.
what if you dont even use dd, just `wget ... | xz -d > foo.img` ?
I'm pretty sure that a simple '>' would do it one bit at a time. Even if it didn't, without a buffer the whole pipe is sensitive to slowdowns.
For example, when flashing directly from the internet, I found that some USB devices occasionally did a flush or something, and that causes a periodic slowdown. With two consecutive buffers, the pipe could keep downloading and extracting for a while in the background.
But without a buffer, the whole pipe would immediately slow down at once, and would rarely ever reach full speed again.
Want to run Minecraft Java on your RPi? Easiest way is with Pi-Apps - just click Install.
https://github.com/Botspot/pi-apps
Pi-Apps also includes Zoom, Visual Studio, Tor browser, Windows 10 theme, and 33 more.
Over 50,000 users!

User avatar
jahboater
Posts: 6511
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: How to download AND extract a .img.xz file?

Tue Oct 13, 2020 1:02 pm

Botspot wrote:
Tue Oct 13, 2020 12:45 pm

I'm pretty sure that a simple '>' would do it one bit at a time.
Its writes with a block size of 8KB (probably BUFSIZ). But likely the speed is limited by the decompress time, as the writes will go straight into memory (the disk cache). Sync must be used afterwards.

You can see the I/O reads and writes for any command with strace, for example:

Code: Select all

strace xz -d Python-3.9.0.tar.xz - >tmp
But you cant see whats going on behind the scenes in the kernel.

Note that cp optimizes prefetch (with POSIX_FADV_SEQUENTIAL ) but dd does not.
Prefetch is key to getting fast overall copy times because it allows the writes to proceed at the same time as the reads, hopefully nearly halving the total time.
That's one reason why commands like cp use a smallish block size (128 KB) because writing cannot commence until the first block read has completed. Giving a huge block size to dd removes much, or all, of the overlap between the reads and the writes!
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

Return to “Beginners”