User avatar
DougieLawson
Posts: 37761
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 6:17 am

ksharindam wrote:
Tue Aug 27, 2019 6:03 am
DougieLawson wrote:
Tue Aug 27, 2019 5:53 am
xz compressed that with -9 and the out of memory killer shot it dead.
How much RAM does your computer have? Is it a Raspberry Pi?
It's a Raspberry Pi 3B.
Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Any DMs sent on Twitter will be answered next month.
All non-medical doctors are on my foes list.

User avatar
bensimmo
Posts: 4339
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 9:41 am

Milliways wrote:
Tue Aug 27, 2019 5:14 am
ksharindam wrote:
Sun Aug 25, 2019 4:23 pm
we know that xz format is more efficient compression method. but the raspbian images are distributed in zip compressed image.

If xz is used at extreme compression level (level 9), much disk space and bandwidth can be saved.
That means Raspbian buster desktop image without recommend softwares, will take around 700mb space instead of 1GB.
So , why zip archive is used instead of xz archive?
I download an image every couple of years to do a fresh install (OK 2 lite & Desktop) I don't care about a few hundred kB or a minute or so download time.

I use gz for backup images.
Similar to my post above,
but then it's been a long time since I used a data limit connection or at least a meaningful one.
My phone yes, but I'd not even bother with that.

Not all countries are the same though. Data cost.

Also I assume RPF/T (not sure which wing hosts it and pay for it) would save in data transfer costs, I assume they have to pay?

I wonder what the torrent usage is to direct download?

User avatar
rpdom
Posts: 16398
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 10:20 am

I've run some tests on various compression methods for the latest full Rasbian Buster release on my Pi 4B 4GB. I rebooted between each test to make sure nothing was cached in RAM.

My Pi 4B is running in open air, no case, with a small home-made heatsink. It idles around 57°C in 25°C ambient temperatures.

I used the default compression options for each method.

Uncompressed file size 6,434,062,336 bytes.

Zip: 17 minutes 45 seconds. 100% of one CPU. Max temperature 65°C. Resulting file size 2,429,397,543 bytes. About 38% of original size.
Gzip: 17 minutes 0 seconds. 100% of one CPU. Max temperature 64°C. Resulting file size 2,429,397,233 bytes. About 38% of original size again.
Bzip2: 34 minutes 0 seconds. 100% of one CPU. Max temperature 67°C. Resulting file size 2,262,242,168 bytes. About 35% of original size.
Xzip: 1 hour 48 minutes 44 seconds. 100% of one CPU. Max temperature 67°C. Resulting file size 1,838,864,556 bytes. About 28.5% of original size.
Xzip with multi-thread: 40 minutes 26 seconds. 100% of all four CPUs. Max temperature 82°C. Resulting file size 1,849,679,760. About 29% of original file size.

Only the last test caused any throttling on my Pi.
Better compression could be achieved with options on each method.
The last xz option would have been faster with better cooling. That is the first time my 4B has been throttled.

I haven't tested decompression times yet.

User avatar
bensimmo
Posts: 4339
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 4:24 pm

It'll be the decompression times that are the important one, how long will we have to wait to get our image.

compression times are only relevant for RPT making the image that one time.
(interesting none the less)

User avatar
rpdom
Posts: 16398
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 5:24 pm

I'll try decompression on the Pi 4 and maybe a Pi Zero tonight and post the results.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6195
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 5:35 pm

Zip was chosen because it's natively supported by all major operating systems. That's a little less relevant now that zip64 support is required, but that's still much more common than xz support.

User avatar
DougieLawson
Posts: 37761
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 8:50 pm

ShiftPlusOne wrote:
Tue Aug 27, 2019 5:35 pm
Zip was chosen because it's natively supported by all major operating systems. That's a little less relevant now that zip64 support is required, but that's still much more common that xz support.
7-zip on Windows can open xz.
Etcher is able to work with all popular compression methods (bz2, xz, zip, gzip, etc.).
Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Any DMs sent on Twitter will be answered next month.
All non-medical doctors are on my foes list.

User avatar
ksharindam
Posts: 156
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Wed Aug 28, 2019 2:11 am

ShiftPlusOne wrote:
Tue Aug 27, 2019 5:35 pm
Zip was chosen because it's natively supported by all major operating systems. That's a little less relevant now that zip64 support is required, but that's still much more common than xz support.
Just a wish, If you provide both zip compressed image and xz compressed (with option -9e) image only for the full desktop image (with recommended software), ii will save our time and bandwidth.
There is no problem with the two other images (426MB and 1.1GB).
But 1.9GB is a very large size for many people (as in many countries data speed is low, and cost is also a factor).

User avatar
bensimmo
Posts: 4339
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Wed Aug 28, 2019 8:20 am

If xz is part of tar, then it is natively supported in Windows10 too.
Only at the command line, I don't think it usable in file explorer.

User avatar
rpdom
Posts: 16398
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Wed Aug 28, 2019 4:10 pm

rpdom wrote:
Tue Aug 27, 2019 10:20 am
I've run some tests on various compression methods for the latest full Rasbian Buster release on my Pi 4B 4GB. I rebooted between each test to make sure nothing was cached in RAM.
Uncompress timings and temperatures (I didn't get CPU usage this time, but it was much lower than compressing. Mostly the times were predictably waiting for the SSD (in this case - I was extracting back to the original file, not writing an image, but it shouldn't make much difference).

These were the compression figures again, just for reference:
Zip: 17 minutes 45 seconds. 100% of one CPU. Max temperature 65°C. Resulting file size 2,429,397,543 bytes. About 38% of original size.
Gzip: 17 minutes 0 seconds. 100% of one CPU. Max temperature 64°C. Resulting file size 2,429,397,233 bytes. About 38% of original size again.
Bzip2: 34 minutes 0 seconds. 100% of one CPU. Max temperature 67°C. Resulting file size 2,262,242,168 bytes. About 35% of original size.
Xzip: 1 hour 48 minutes 44 seconds. 100% of one CPU. Max temperature 67°C. Resulting file size 1,838,864,556 bytes. About 28.5% of original size.
Xzip with multi-thread: 40 minutes 26 seconds. 100% of all four CPUs. Max temperature 82°C. Resulting file size 1,849,679,760. About 29% of original file size.

and decompression:
Zip: 9 minutes 41 seconds. Max temperature 68°C
Gzip: 7 minutes 15 seconds. Max temperature 71°C
Bzip2: 18 minutes 48 seconds. Max temperature 75°C
Xzip: 8 minutes 27 seconds. Max temperature 75°C
Xzip multipass-thread: 8 minutes 5 seconds. Max temperature 70°C

Interestingly gzip is best for decompression in this case.

Notes:
I only used default compression level for each method. Higher compression could be achieved, but would take a lot longer to compress and longer to decompress on lower performance machines.
As I think I mentioned originally I'm using a Pi 4B 4GB in open air, ambient around 25°C with a small home-made heatsink (10mmx10mmx18mm high). USB3.0 SATA adaptor with 120GB SSD. Installed OS is Raspbian Lite, fully updated when I started the compression tests.

Return to “Raspbian”