Need to create Pi image to send to user

[Bosse_B]

I am setting up an RPi3 using the latest Pi-OS (previously Raspbian) so it can be used in a prototyping environment at a site a long ways away from here.
The plan is to configure an Rpi3 here fully for the job that needs to be done and then create an image file of the SDcard, which is as small as possible so it can be transfered over the Internet to the target location.
There a tech will pick up the file and burn it to an SDcard using a tool like Balena Etcher and then he will put it into the RPi3 in the prototype.

For this scheme I need the file to be as small as possible and able to be expanded onto a smaller SDcard than the original, just like the Pi-OS downloads work.

How can this be done?

I have read the following threads but they seem not to cover this request:
No Raspberry Pi imager for Raspberry Pi
Raspberry Pi Imager for Raspberry Pi OS

So to clarify:
I want to clone the running configured RPi SDcard to an image file which internally specifies as small an image as possible but when first started on the target RPi hardware it will auto-expand to use the full size of the SDcard it has been burned to.
Is this at all possible?

[pcmanbob]

Yes it is possible.

I do it using pishrink https://github.com/Drewsif/PiShrink

My normal process would be to use windsk32imager to make a copy of the SD card,

Then transfer this image on to an old desktop PC running Ubuntu server 16.4 ,

Then I run pishrink against the image file producing a copy that is as small as possible but that will still boot and auto expand.

I recommend testing the image before sending it to the other location just to be sure all is well.

You can of course do the whole process on a PC running Linux it just that my card reader is installed in my windows PC and I use that to write all my SD cards with images.

[Bosse_B]

Thanks for the link!
Given that this is a shellscript, could it also run on an RPi4?

You say you use the windsk32imager program on Windows to create the initial (too big) image file, is this a typo so it should really be Win32DiskImager you use?
In that case I have previously used that but found it to produce these really big files that could barely be moved.
In my case I will wind up with a 32+GB file after making an image of an SDcard that is using only a small fraction of that space.
Is there not some other tool that can be used to produce the right size image directly from the source SDcard?

[pcmanbob]

Given that this is a shellscript, could it also run on an RPi4?

I have never had an SD card that could hold the running OS and the image file that I wanted to shrink so have never tested pishrink on a pi.

Yes I meant Win32DiskImager , yes it does produce a byte for byte image of the SD card, but never had problems moving them as all my PC's are on a gigabit LAN , I even run the Linux PC headless doing everything via ssh just like I would with my pi's.

Is there not some other tool that can be used to produce the right size image directly from the source SDcard?

Not that I can think of off the top of my head, you could use the SD card copier to copy your 32GB SD card to a smaller SD card directly on the pi, assuming you are using the desktop version of raspberry pi OS and then make a image of the smaller SD card for pishrink to shrink down , but that would add yet another step.

You could also just make the image of the SD card directly on a PC running linux and then run pishrink with no need to move any files.

[hippy]

An alternative is to create a bootable SD Card then simply post or courier it to the recipient.

That may be easier all round, for you and the client.

[nliviu]

Backup SD card to .IMG file

[Bosse_B]

Backup SD card to .IMG file

PERFECT!
Seems like it is all that I need, except I have to compress the final image file, not a big deal.
Now I will have to make a test, so I must find a target disk to attach to the RPi3 where I am configuring the system...
Not enough bandwidth on the internal WiFi network to use a share here.

[Bosse_B]

Did a test using an USB disk I had available.
It is a 500 GB NTFS drive with half of the space free.
So I mounted it into /mnt/nasdisk and ran the following commands:

Code: Select all

$ sudo su
# mount /dev/sda1 /mnt/nasdisk
# time imgclone -d /mnt/nasdisk/RPI/testimg.img

After 40 seconds it finished, but there were a lot of error messages on screen.
Still the testimg.img file was created and sized about 5.6 GB:

Code: Select all

i# ls -la /mnt/nasdisk/RPI/
total 21192
drwxrwxrwx 1 root root          0 Aug  4 19:31 .
drwxrwxrwx 1 root root       4096 Aug  4 19:21 ..
-rwxrwxrwx 1 root root 5694170112 Aug  4 19:31 testimg.img

I just re-ran it again with this result:

Code: Select all

# time imgclone -d /mnt/nasdisk/RPI/testimg2.img
----    Raspberry Pi clone to image V1.1    ---
-----------------------------------------------
---- DO NOT CHANGE FILES ON YOUR SD CARD    ---
---- WHILE THE BACKUP PROGRAM IS RUNNING    ---
---- THE DESTINATION .IMG FILE MUST BE      ---
---- ON AN EXTERNAL STORAGE / NETWORK SHARE ---
-----------------------------------------------
Cloning /dev/mmcblk0 to /mnt/nasdisk/RPI/testimg2.img
uuid | cut -f1 -d-
mktemp -d
mktemp -d
-----------------------------------------------
----    READING PARTITIONS               ------
-----------------------------------------------
Partition 1 start: 8192 end: 532479 ptype:primary ftype:fat32 flags: lba
.Partition 2 start: 532480 end: 61120511 ptype:primary ftype:ext4 flags:
.partprobe
Last partition starts at 272629760 bytes.
mount /dev/mmcblk0p2 /tmp/tmp.fQEPvuAm6T
df /tmp/tmp.fQEPvuAm6T | tail -n 1 | tr -s " " " " | cut -d ' ' -f 2
df /tmp/tmp.fQEPvuAm6T | tail -n 1 | tr -s " " " " | cut -d ' ' -f 4
Used size of last partition is 5350297600 bytes.
umount /tmp/tmp.fQEPvuAm6T
Required size for destination image: 5745871872 bytes
-----------------------------------------------
----    ALLOCATING SPACE FOR .IMG FILE   ------
-----------------------------------------------
touch "/mnt/nasdisk/RPI/testimg2.img"
df "/mnt/nasdisk/RPI/testimg2.img" | tail -n 1 | tr -s " " " " | cut -d ' ' -f 1
/dev/sda1
df --output=avail -B 1 "/mnt/nasdisk/RPI/testimg2.img" | tail -n 1
123168919552
truncate --size 5745871872 "/mnt/nasdisk/RPI/testimg2.img"
-----------------------------------------------
----  CREATING DISK DEVICE FOR .IMG FILE ------
-----------------------------------------------
losetup --show -f "/mnt/nasdisk/RPI/testimg2.img"
dd if=/dev/zero of=/dev/loop0 bs=512 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.0711845 s, 7.2 kB/s
Creating FAT on /dev/loop0.
parted -s /dev/loop0 mklabel msdos
-----------------------------------------------
----    CREATING PARTITIONS PLEASE WAIT  ------
-----------------------------------------------
parted -s /dev/loop0 mkpart primary fat32 8192s 532479s
partprobe
lsblk -o name,uuid /dev/mmcblk0 | grep mmcblk0p1 | tr -s " " | cut -d " " -f 2
lsblk -o name,label /dev/mmcblk0 | grep mmcblk0p1 | tr -s " " | cut -d " " -f 2
blkid /dev/mmcblk0 | rev | cut -f 2 -d ' ' | rev | cut -f 2 -d \"
mkfs.fat -F 32 -i 70A28001 /dev/loop0p1
mkfs.fat 4.1 (2017-01-24)
fatlabel /dev/loop0p1 boot
fatlabel: warning - lowercase labels might not work properly with DOS or Windows
echo "x
i
0x047f45e6
r
w
" | fdisk /dev/loop0

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help):
Expert command (m for help):
Enter the new disk identifier:
Disk identifier changed from 0x14a05a21 to 0x047f45e6.

Expert command (m for help):
Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

parted -s /dev/loop0 -- mkpart primary ext4 532480s -1s
partprobe
lsblk -o name,uuid /dev/mmcblk0 | grep mmcblk0p2 | tr -s " " | cut -d " " -f 2
lsblk -o name,label /dev/mmcblk0 | grep mmcblk0p2 | tr -s " " | cut -d " " -f 2
blkid /dev/mmcblk0 | rev | cut -f 2 -d ' ' | rev | cut -f 2 -d \"
mkfs.ext4 -F -U a1fafd2b-1ef0-4fe8-8ac1-ad33bbb48642 /dev/loop0p2
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 1336240 4k blocks and 334560 inodes
Filesystem UUID: a1fafd2b-1ef0-4fe8-8ac1-ad33bbb48642
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
mkfs.ext4: Input/output error while writing out and closing file system
Writing superblocks and filesystem accounting information: mkfs.ext4 -F /dev/loop0p2
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 1336240 4k blocks and 334560 inodes
Filesystem UUID: 3938956c-593c-49a7-8db5-7ef796a4d684
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
mkfs.ext4: Input/output error while writing out and closing file system
Could not create file system.
Writing superblocks and filesystem accounting information:
real    0m28.034s
user    0m0.612s
sys     0m3.324s

So I just wonder based on the errors if I must use some other kind of USB disk to create the image on?
Can an NTFS drive be used at all?

[Bosse_B]

So I did the following:
- Put an SDcard that holds an RPi image in a USB carrier and inserted it
- Then examined the state using lsblk and found the card registered as sdb with the root at sdb2 (ext4 filesystem)
- Mounted sdb2 as /mnt/nasdisk
- Navigated to /mnt/nasdisk/ and created a new dir (with sudo) /mnt/nasdisk/backup
- Moved to this directory and issued:

Code: Select all

# time imgclone -d testimage.img

This command ran for 12m9s and the final output on screen was:

Code: Select all

-----------------------------------------------
----    COPYING FILES PLEASE WAIT        ------
-----------------------------------------------
cp -ax /tmp/tmp.OPMU9VOFGT/. /tmp/tmp.R0YI7dDyR5/.
umount /tmp/tmp.R0YI7dDyR5
umount /tmp/tmp.OPMU9VOFGT
parted -s /dev/loop1 set 2 lba off
losetup -d /dev/loop1
Backup completed!  <== Seems like a success message!
rm /dev/loop1p1
rm /dev/loop1p2

real    12m9.372s  <== Believable execution time!
user    0m4.501s
sys     1m17.113s

The produced file is 5.6 GB in size.
Right now I am compressing it (# gzip -k testimage.img)

It looks like one should use an ext4 file system on which to place the output img file....

But gzip-ing it failed anyway:

Code: Select all

root@rpi3-agissr:/mnt/nasdisk/backup# gzip -k testimage.img
gzip: testimage.img: Input/output error

And after it failed I can no longer access the disk to check the file created.
The mount seems to have evaporated and I cannot recreate it.

[nliviu]

No more space on the backup sd?

After patching the imgclone repo, I've run it to create a backup of my ssd onto a nfs mount and it worked without any issues.

LE. My PR has been merged.

[Bosse_B]

No more space on the backup sd?

After patching the imgclone repo, I've run it to create a backup of my ssd onto a nfs mount and it worked without any issues.

LE. My PR has been merged.

I don't think it is a space problem, it could be the SDcard USB carrier (a Canakit microsd reader) that is acting up. Now it has completely stopped working...

Using another SDcard carrier and repeating the process resulted in a 5.6GB file I could copy over to the original Pi SDcard before unmounting the target disk.
gzip compressed it to 1.2 GB, which is OK for my transfer to the project site.

I have pulled the changes and rebuilt the imgclone program now. Thanks!

Question:
What are the changes you have done?

[Bosse_B]

FOLLOW-UP
Today I created a data only SDcard using gparted in Pi-OS.
It is this I will use to host the image file from the RPi.
I have also made a lot of installations on the RPi3 such that it will be a lot bigger this time.
The RPi3 SDcard is 32 GB in size.

Here are my results from a cycle:

Code: Select all

pi@rpi3-agissr:/media/pi/datastore/backup $ time sudo imgclone -d agissr_20200806.img
----    Raspberry Pi clone to image V1.1    ---
-----------------------------------------------
....

real    42m19.203s
user    0m7.990s
sys     2m25.972s
pi@rpi3-agissr:/media/pi/datastore/backup $ ll
total 6866128
drwxr-xr-x 2 root root       4096 Aug  6 16:04 .
drwxr-xr-x 4 root root       4096 Aug  6 16:01 ..
-rw-r--r-- 1 root root 8600406528 Aug  6 16:46 agissr_20200806.img
pi@rpi3-agissr:/media/pi/datastore/backup $ time sudo gzip agissr_20200806.img

real    25m45.023s
user    22m16.613s
sys     0m49.643s
pi@rpi3-agissr:/media/pi/datastore/backup $ ll
total 1790272
drwxr-xr-x 2 root root       4096 Aug  6 18:35 .
drwxr-xr-x 4 root root       4096 Aug  6 16:01 ..
-rw-r--r-- 1 root root 1833224600 Aug  6 16:46 agissr_20200806.img.gz

So the img file creation takes 42+ minutes to make an 8.6 GB img file.
Then another 25+ minutes to compress the file using gzip to 1.8 GB

[Bosse_B]

Final report...
Over the week-end I used the imgclone function to create an image of the prototype RPi3B device I have configured here.
I used an NFS share on my Ubuntu 18 home server as the image target and it worked just fine, took 30 min.
Then on Ubuntu I gzip-ed the img file (another 15 min) and moved the gz file into my webserver area.
Next I connected by VPN to the remote location and logged on to the server over there and used wget to retrieve the file.

Then yesterday a tech on location (in Texas) used the image file and Balena Etcher to write an SDcard which he then used on the RPi3 intended for the test equipment.
It worked just fine and today he will install it in the test equipment.

The imgclone utility is REALLY a useful tool!

Thanks a bunch for steering me towards it!