User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

What is the best way to create compressed disk image?

Thu Mar 05, 2020 11:45 pm

Hi,

What is the best way to create a compressed disk image which would not exceed 2GB? The requirement for 2GB file comes from the Github which allows to host files 2GB max. You can break it into chunks but that overcomplicates the task for people who will use that disk image.

My current process includes too many steps and I’m just wondering if it’s possible to simplify it somehow:
- Use 8GB SD card, install all required software on that card on Pi
- Remove card from Pi and insert it into my Windows 10 desktop
- Using Win32 Disk Imager create ‘img’ file
- Insert another 16GB SD card with ‘pishrink’ program into Pi
- Copy ~8GB img file from Windows to Pi using WinScp
- Run ‘pishrink’ which will compress ~8GB img file into ~4GB file
- Copy shrinked file back to Windows machine using WinScp
- Compress ~4GB shrinked file using zip. The file becomes about 1.8 GB
- Copy 1.8GB image file to Github

Is there any way to improve/simplify the process? It takes too much time right now.
Thank you!
Last edited by peppy.player on Fri Mar 06, 2020 6:23 pm, edited 1 time in total.

User avatar
DougieLawson
Posts: 40138
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: What is the best way to create compressed disk image?

Thu Mar 05, 2020 11:52 pm

Use parted or gparted to shrink your system down to the minimum possible size. You'll need your RPi booted from some other SDCard and the filesystem you want to shrink in a USB reader.
Steal the filesystem resize code (at first boot) stuff from Raspbian. So you can expand it back to full size.
Write an image file using a Linux system (you've got one of those on your RPi). Don't write the unused space beyond your shrunken partition.
Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All fake doctors are on my foes list.

Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 12:02 am

Thank you for reply!
I'm not sure where I can save time using your approach? I still need to create image file on my Windows machine and copy it to USB flash drive. Though probably copying to USB flash drive is faster than copying to Pi over SSH.

RonR
Posts: 1682
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 12:14 am

peppy.player wrote:
Thu Mar 05, 2020 11:45 pm
What is the best way to create a compressed disk image which would not exceed 2GB?

Have you explored Image File Utilities

User avatar
DougieLawson
Posts: 40138
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 12:18 am

I can't work out what you think you need on Windows. Everything can be done on that nice little Linux system that you own, including writing SDCards or USB sticks.
Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All fake doctors are on my foes list.

Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 12:27 am

DougieLawson wrote:
Fri Mar 06, 2020 12:18 am
I can't work out what you think you need on Windows. Everything can be done on that nice little Linux system that you own, including writing SDCards or USB sticks.
How the system which I need to shrink will appear on USB drive?

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 12:32 am

RonR wrote:
Fri Mar 06, 2020 12:14 am
peppy.player wrote:
Thu Mar 05, 2020 11:45 pm
What is the best way to create a compressed disk image which would not exceed 2GB?

Have you explored Image File Utilities
I use 'shrink' utility. Though I don't think it's from your library.

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 6:21 pm

I have outlined my current procedure in the first post. Is it possible to do the same for any other approach which could save time? Anyone who does that in a different way please post your steps here. Thanks in advance!

Paul Hutch
Posts: 561
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 9:03 pm

I use basically the same process as you listed except I avoid moving the files and cards around by doing it all on a spare Pi running desktop Raspbian.

The spare Pi has a uSD card reader and extra USB storage for the files, and I use dd to do the read/write and 7Zip for the final compression.

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 9:17 pm

Paul Hutch wrote:
Fri Mar 06, 2020 9:03 pm
I use basically the same process as you listed except I avoid moving the files and cards around by doing it all on a spare Pi running desktop Raspbian.

The spare Pi has a uSD card reader and extra USB storage for the files, and I use dd to do the read/write and 7Zip for the final compression.
That sounds interesting. Paul, could you describe the whole process? Thanks in advance!

Paul Hutch
Posts: 561
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 9:44 pm

I plagiarized your list as a starting point :-)

- Use 8GB SD card, install all required software on that card on Pi
- Remove card from Pi and insert it into my desktop Pi's card reader
- Use dd to create an ‘img’ file on the USB storage (HDD or thumb drive depending on which extra Pi I'm using)
- Run ‘pishrink’ on the created image
- Compress the shrunken file using 7Zip.
- Copy file to Internet

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Fri Mar 06, 2020 9:59 pm

Paul Hutch wrote:
Fri Mar 06, 2020 9:44 pm
I plagiarized your list as a starting point :-)

- Use 8GB SD card, install all required software on that card on Pi
- Remove card from Pi and insert it into my desktop Pi's card reader
- Use dd to create an ‘img’ file on the USB storage (HDD or thumb drive depending on which extra Pi I'm using)
- Run ‘pishrink’ on the created image
- Compress the shrunken file using 7Zip.
- Copy file to Internet
Thank you! This approach can really save time by eliminating the time for transferring files back and forth. I believe instead of the second Pi the same one could be used but with different SD card, right?

Paul Hutch
Posts: 561
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: What is the best way to create compressed disk image?

Sat Mar 07, 2020 12:21 pm

Yes you could just use the same Pi with a second card setup for the image copy process. If you used an extra large second card you can even skip the external USB storage for storing the large disk images.

ConfuSomu
Posts: 42
Joined: Tue Dec 04, 2018 8:47 am
Location: Canada

Re: What is the best way to create compressed disk image?

Sun Mar 08, 2020 5:33 pm

peppy.player wrote:
Thu Mar 05, 2020 11:45 pm
- Compress ~4GB shrinked file using zip. The file becomes about 1.8 GB
Compressing the image using the bzip2 compression algorithm is more efficient than using zip (with DEFLATE compression). You can compress a file with the bzip2 compression algorithm using the 7-Zip compression utility. Make sure to choose bzip2 in the interface.

Hope that helps.
If you are low on RAM, make sure to use zram! ;)
I found & use this zram setup script: https://github.com/novaspirit/rpi_zram
A post about zram: t=207304
Make sure to disable dphys-swapfile and mount the tmp folders in RAM. Also learn about SysRq!

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Sun Mar 08, 2020 9:14 pm

ConfuSomu wrote:
Sun Mar 08, 2020 5:33 pm
peppy.player wrote:
Thu Mar 05, 2020 11:45 pm
- Compress ~4GB shrinked file using zip. The file becomes about 1.8 GB
Compressing the image using the bzip2 compression algorithm is more efficient than using zip (with DEFLATE compression). You can compress a file with the bzip2 compression algorithm using the 7-Zip compression utility. Make sure to choose bzip2 in the interface.

Hope that helps.
Thank you for advice. Though that assumes that everyone who will use that compressed disk image should also install 7-zip.

User avatar
Greg Erskine
Posts: 162
Joined: Sat Sep 15, 2012 4:20 am

Re: What is the best way to create compressed disk image?

Sun Mar 08, 2020 10:17 pm

hi peppy.player,

I think you need to use Linux to script the creation of your image. You need to make it a one step process for your sanity. :D

We compress pCP to a zip format compatible with Windows 10.

regards
Greg
* Raspberry Pi is a trademark of the Raspberry Pi Foundation

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Mon Mar 09, 2020 1:25 am

Next time I'll try slightly modified approach suggested by Paul:

- Connect USB card reader to Pi
- Use 8GB SD card inserted into Pi, install all required software on that card on Pi
- Remove card from Pi and insert it into card reader
- Insert another 16GB SD card with ‘pishrink’ program into Pi
- Use dd to create an ‘img’ file from 8BG card inserted into reader on the 16GB card inserted to Pi
- Run ‘pishrink’ which will compress ~8GB img file into ~4GB file
- Zip shrunken file ~4GB -> 1.8GB

3 steps in this procedure can be scripted: dd, pishrink, zip.

Zwartoog
Posts: 8
Joined: Sun Mar 08, 2020 5:27 pm

Re: What is the best way to create compressed disk image?

Mon Mar 09, 2020 11:26 am

I am not sure why you keep on moving back-and-forth between Windows and the Pi.

I assume this is a process you do yourself to prepare a suitable image to prepare for the generic audience?

I would do something like this:
- by a USB-SD card reader
- prepare your distribution on SD card 1, i.e. SDPEPPY
- now boot your Pi with SD card 2, i.e. SDRASPBIAN
- insert SDPEPPY in USB-SD
- use dd to create an image from SDPEPPY, write this to SDRASPBIAN (or a generic USB-key, or network storage)
- use phishrink/zip/whatever you need on this image

Aside from swapping the SD cards to boot in Raspbian, you can script all the other steps.

User avatar
peppy.player
Posts: 317
Joined: Mon Mar 07, 2016 6:10 am

Re: What is the best way to create compressed disk image?

Mon Mar 09, 2020 2:48 pm

Zwartoog wrote:
Mon Mar 09, 2020 11:26 am
I am not sure why you keep on moving back-and-forth between Windows and the Pi.

I assume this is a process you do yourself to prepare a suitable image to prepare for the generic audience?

I would do something like this:
- by a USB-SD card reader
- prepare your distribution on SD card 1, i.e. SDPEPPY
- now boot your Pi with SD card 2, i.e. SDRASPBIAN
- insert SDPEPPY in USB-SD
- use dd to create an image from SDPEPPY, write this to SDRASPBIAN (or a generic USB-key, or network storage)
- use phishrink/zip/whatever you need on this image

Aside from swapping the SD cards to boot in Raspbian, you can script all the other steps.
I suspect you didn't see my last post. Thank you anyway.

Zwartoog
Posts: 8
Joined: Sun Mar 08, 2020 5:27 pm

Re: What is the best way to create compressed disk image?

Mon Mar 09, 2020 9:34 pm

Ouch, that's embarrassing :oops:

Return to “Advanced users”