Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Need to create Pi image to send to user

Mon Aug 03, 2020 8:02 pm

I am setting up an RPi3 using the latest Pi-OS (previously Raspbian) so it can be used in a prototyping environment at a site a long ways away from here.
The plan is to configure an Rpi3 here fully for the job that needs to be done and then create an image file of the SDcard, which is as small as possible so it can be transfered over the Internet to the target location.
There a tech will pick up the file and burn it to an SDcard using a tool like Balena Etcher and then he will put it into the RPi3 in the prototype.

For this scheme I need the file to be as small as possible and able to be expanded onto a smaller SDcard than the original, just like the Pi-OS downloads work.

How can this be done?

I have read the following threads but they seem not to cover this request:
No Raspberry Pi imager for Raspberry Pi
Raspberry Pi Imager for Raspberry Pi OS

So to clarify:
I want to clone the running configured RPi SDcard to an image file which internally specifies as small an image as possible but when first started on the target RPi hardware it will auto-expand to use the full size of the SDcard it has been burned to.
Is this at all possible?
Bo Berglund
Sweden

pcmanbob
Posts: 10178
Joined: Fri May 31, 2013 9:28 pm
Location: Mansfield UK

Re: Need to create Pi image to send to user

Mon Aug 03, 2020 10:41 pm

Yes it is possible.

I do it using pishrink https://github.com/Drewsif/PiShrink

My normal process would be to use windsk32imager to make a copy of the SD card,

Then transfer this image on to an old desktop PC running Ubuntu server 16.4 ,

Then I run pishrink against the image file producing a copy that is as small as possible but that will still boot and auto expand.

I recommend testing the image before sending it to the other location just to be sure all is well.

You can of course do the whole process on a PC running Linux it just that my card reader is installed in my windows PC and I use that to write all my SD cards with images.
We want information… information… information........................no information no help
The use of crystal balls & mind reading are not supported

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 5:42 am

Thanks for the link!
Given that this is a shellscript, could it also run on an RPi4?

You say you use the windsk32imager program on Windows to create the initial (too big) image file, is this a typo so it should really be Win32DiskImager you use?
In that case I have previously used that but found it to produce these really big files that could barely be moved.
In my case I will wind up with a 32+GB file after making an image of an SDcard that is using only a small fraction of that space.
Is there not some other tool that can be used to produce the right size image directly from the source SDcard?
Bo Berglund
Sweden

pcmanbob
Posts: 10178
Joined: Fri May 31, 2013 9:28 pm
Location: Mansfield UK

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 8:28 am

Given that this is a shellscript, could it also run on an RPi4?
I have never had an SD card that could hold the running OS and the image file that I wanted to shrink so have never tested pishrink on a pi.

Yes I meant Win32DiskImager , yes it does produce a byte for byte image of the SD card, but never had problems moving them as all my PC's are on a gigabit LAN , I even run the Linux PC headless doing everything via ssh just like I would with my pi's.
Is there not some other tool that can be used to produce the right size image directly from the source SDcard?
Not that I can think of off the top of my head, you could use the SD card copier to copy your 32GB SD card to a smaller SD card directly on the pi, assuming you are using the desktop version of raspberry pi OS and then make a image of the smaller SD card for pishrink to shrink down , but that would add yet another step.

You could also just make the image of the SD card directly on a PC running linux and then run pishrink with no need to move any files.
We want information… information… information........................no information no help
The use of crystal balls & mind reading are not supported

hippy
Posts: 8593
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 9:37 am

An alternative is to create a bootable SD Card then simply post or courier it to the recipient.

That may be easier all round, for you and the client.


Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 3:48 pm

nliviu wrote:
Tue Aug 04, 2020 11:26 am
Backup SD card to .IMG file
PERFECT! :D :D :D
Seems like it is all that I need, except I have to compress the final image file, not a big deal.
Now I will have to make a test, so I must find a target disk to attach to the RPi3 where I am configuring the system...
Not enough bandwidth on the internal WiFi network to use a share here.
Bo Berglund
Sweden

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 6:55 pm

Did a test using an USB disk I had available.
It is a 500 GB NTFS drive with half of the space free.
So I mounted it into /mnt/nasdisk and ran the following commands:

Code: Select all

$ sudo su
# mount /dev/sda1 /mnt/nasdisk
# time imgclone -d /mnt/nasdisk/RPI/testimg.img
After 40 seconds it finished, but there were a lot of error messages on screen.
Still the testimg.img file was created and sized about 5.6 GB:

Code: Select all

i# ls -la /mnt/nasdisk/RPI/
total 21192
drwxrwxrwx 1 root root          0 Aug  4 19:31 .
drwxrwxrwx 1 root root       4096 Aug  4 19:21 ..
-rwxrwxrwx 1 root root 5694170112 Aug  4 19:31 testimg.img
I just re-ran it again with this result:

Code: Select all

# time imgclone -d /mnt/nasdisk/RPI/testimg2.img
----    Raspberry Pi clone to image V1.1    ---
-----------------------------------------------
---- DO NOT CHANGE FILES ON YOUR SD CARD    ---
---- WHILE THE BACKUP PROGRAM IS RUNNING    ---
---- THE DESTINATION .IMG FILE MUST BE      ---
---- ON AN EXTERNAL STORAGE / NETWORK SHARE ---
-----------------------------------------------
Cloning /dev/mmcblk0 to /mnt/nasdisk/RPI/testimg2.img
uuid | cut -f1 -d-
mktemp -d
mktemp -d
-----------------------------------------------
----    READING PARTITIONS               ------
-----------------------------------------------
Partition 1 start: 8192 end: 532479 ptype:primary ftype:fat32 flags: lba
.Partition 2 start: 532480 end: 61120511 ptype:primary ftype:ext4 flags:
.partprobe
Last partition starts at 272629760 bytes.
mount /dev/mmcblk0p2 /tmp/tmp.fQEPvuAm6T
df /tmp/tmp.fQEPvuAm6T | tail -n 1 | tr -s " " " " | cut -d ' ' -f 2
df /tmp/tmp.fQEPvuAm6T | tail -n 1 | tr -s " " " " | cut -d ' ' -f 4
Used size of last partition is 5350297600 bytes.
umount /tmp/tmp.fQEPvuAm6T
Required size for destination image: 5745871872 bytes
-----------------------------------------------
----    ALLOCATING SPACE FOR .IMG FILE   ------
-----------------------------------------------
touch "/mnt/nasdisk/RPI/testimg2.img"
df "/mnt/nasdisk/RPI/testimg2.img" | tail -n 1 | tr -s " " " " | cut -d ' ' -f 1
/dev/sda1
df --output=avail -B 1 "/mnt/nasdisk/RPI/testimg2.img" | tail -n 1
123168919552
truncate --size 5745871872 "/mnt/nasdisk/RPI/testimg2.img"
-----------------------------------------------
----  CREATING DISK DEVICE FOR .IMG FILE ------
-----------------------------------------------
losetup --show -f "/mnt/nasdisk/RPI/testimg2.img"
dd if=/dev/zero of=/dev/loop0 bs=512 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.0711845 s, 7.2 kB/s
Creating FAT on /dev/loop0.
parted -s /dev/loop0 mklabel msdos
-----------------------------------------------
----    CREATING PARTITIONS PLEASE WAIT  ------
-----------------------------------------------
parted -s /dev/loop0 mkpart primary fat32 8192s 532479s
partprobe
lsblk -o name,uuid /dev/mmcblk0 | grep mmcblk0p1 | tr -s " " | cut -d " " -f 2
lsblk -o name,label /dev/mmcblk0 | grep mmcblk0p1 | tr -s " " | cut -d " " -f 2
blkid /dev/mmcblk0 | rev | cut -f 2 -d ' ' | rev | cut -f 2 -d \"
mkfs.fat -F 32 -i 70A28001 /dev/loop0p1
mkfs.fat 4.1 (2017-01-24)
fatlabel /dev/loop0p1 boot
fatlabel: warning - lowercase labels might not work properly with DOS or Windows
echo "x
i
0x047f45e6
r
w
" | fdisk /dev/loop0

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help):
Expert command (m for help):
Enter the new disk identifier:
Disk identifier changed from 0x14a05a21 to 0x047f45e6.

Expert command (m for help):
Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

parted -s /dev/loop0 -- mkpart primary ext4 532480s -1s
partprobe
lsblk -o name,uuid /dev/mmcblk0 | grep mmcblk0p2 | tr -s " " | cut -d " " -f 2
lsblk -o name,label /dev/mmcblk0 | grep mmcblk0p2 | tr -s " " | cut -d " " -f 2
blkid /dev/mmcblk0 | rev | cut -f 2 -d ' ' | rev | cut -f 2 -d \"
mkfs.ext4 -F -U a1fafd2b-1ef0-4fe8-8ac1-ad33bbb48642 /dev/loop0p2
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 1336240 4k blocks and 334560 inodes
Filesystem UUID: a1fafd2b-1ef0-4fe8-8ac1-ad33bbb48642
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
mkfs.ext4: Input/output error while writing out and closing file system
Writing superblocks and filesystem accounting information: mkfs.ext4 -F /dev/loop0p2
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 1336240 4k blocks and 334560 inodes
Filesystem UUID: 3938956c-593c-49a7-8db5-7ef796a4d684
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
mkfs.ext4: Input/output error while writing out and closing file system
Could not create file system.
Writing superblocks and filesystem accounting information:
real    0m28.034s
user    0m0.612s
sys     0m3.324s
So I just wonder based on the errors if I must use some other kind of USB disk to create the image on?
Can an NTFS drive be used at all?
Bo Berglund
Sweden

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Tue Aug 04, 2020 9:46 pm

So I did the following:
- Put an SDcard that holds an RPi image in a USB carrier and inserted it
- Then examined the state using lsblk and found the card registered as sdb with the root at sdb2 (ext4 filesystem)
- Mounted sdb2 as /mnt/nasdisk
- Navigated to /mnt/nasdisk/ and created a new dir (with sudo) /mnt/nasdisk/backup
- Moved to this directory and issued:

Code: Select all

# time imgclone -d testimage.img
This command ran for 12m9s and the final output on screen was:

Code: Select all

-----------------------------------------------
----    COPYING FILES PLEASE WAIT        ------
-----------------------------------------------
cp -ax /tmp/tmp.OPMU9VOFGT/. /tmp/tmp.R0YI7dDyR5/.
umount /tmp/tmp.R0YI7dDyR5
umount /tmp/tmp.OPMU9VOFGT
parted -s /dev/loop1 set 2 lba off
losetup -d /dev/loop1
Backup completed!  <== Seems like a success message!
rm /dev/loop1p1
rm /dev/loop1p2

real    12m9.372s  <== Believable execution time!
user    0m4.501s
sys     1m17.113s
The produced file is 5.6 GB in size.
Right now I am compressing it (# gzip -k testimage.img)

It looks like one should use an ext4 file system on which to place the output img file....

But gzip-ing it failed anyway:

Code: Select all

root@rpi3-agissr:/mnt/nasdisk/backup# gzip -k testimage.img
gzip: testimage.img: Input/output error
And after it failed I can no longer access the disk to check the file created.
The mount seems to have evaporated and I cannot recreate it.
Bo Berglund
Sweden

nliviu
Posts: 51
Joined: Tue Jun 23, 2020 1:24 pm
Location: Romania

Re: Need to create Pi image to send to user

Wed Aug 05, 2020 6:46 pm

No more space on the backup sd?

After patching the imgclone repo, I've run it to create a backup of my ssd onto a nfs mount and it worked without any issues.

LE. My PR has been merged.

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Wed Aug 05, 2020 7:50 pm

nliviu wrote:
Wed Aug 05, 2020 6:46 pm
No more space on the backup sd?

After patching the imgclone repo, I've run it to create a backup of my ssd onto a nfs mount and it worked without any issues.

LE. My PR has been merged.
I don't think it is a space problem, it could be the SDcard USB carrier (a Canakit microsd reader) that is acting up. Now it has completely stopped working... :evil:

Using another SDcard carrier and repeating the process resulted in a 5.6GB file I could copy over to the original Pi SDcard before unmounting the target disk.
gzip compressed it to 1.2 GB, which is OK for my transfer to the project site.

I have pulled the changes and rebuilt the imgclone program now. Thanks!

Question:
What are the changes you have done?
Bo Berglund
Sweden

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Thu Aug 06, 2020 4:52 pm

FOLLOW-UP
Today I created a data only SDcard using gparted in Pi-OS.
It is this I will use to host the image file from the RPi.
I have also made a lot of installations on the RPi3 such that it will be a lot bigger this time.
The RPi3 SDcard is 32 GB in size.

Here are my results from a cycle:

Code: Select all

pi@rpi3-agissr:/media/pi/datastore/backup $ time sudo imgclone -d agissr_20200806.img
----    Raspberry Pi clone to image V1.1    ---
-----------------------------------------------
....

real    42m19.203s
user    0m7.990s
sys     2m25.972s
pi@rpi3-agissr:/media/pi/datastore/backup $ ll
total 6866128
drwxr-xr-x 2 root root       4096 Aug  6 16:04 .
drwxr-xr-x 4 root root       4096 Aug  6 16:01 ..
-rw-r--r-- 1 root root 8600406528 Aug  6 16:46 agissr_20200806.img
pi@rpi3-agissr:/media/pi/datastore/backup $ time sudo gzip agissr_20200806.img

real    25m45.023s
user    22m16.613s
sys     0m49.643s
pi@rpi3-agissr:/media/pi/datastore/backup $ ll
total 1790272
drwxr-xr-x 2 root root       4096 Aug  6 18:35 .
drwxr-xr-x 4 root root       4096 Aug  6 16:01 ..
-rw-r--r-- 1 root root 1833224600 Aug  6 16:46 agissr_20200806.img.gz
So the img file creation takes 42+ minutes to make an 8.6 GB img file.
Then another 25+ minutes to compress the file using gzip to 1.8 GB
Bo Berglund
Sweden

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Tue Aug 11, 2020 4:21 pm

Final report...
Over the week-end I used the imgclone function to create an image of the prototype RPi3B device I have configured here.
I used an NFS share on my Ubuntu 18 home server as the image target and it worked just fine, took 30 min.
Then on Ubuntu I gzip-ed the img file (another 15 min) and moved the gz file into my webserver area.
Next I connected by VPN to the remote location and logged on to the server over there and used wget to retrieve the file.

Then yesterday a tech on location (in Texas) used the image file and Balena Etcher to write an SDcard which he then used on the RPi3 intended for the test equipment.
It worked just fine and today he will install it in the test equipment.

The imgclone utility is REALLY a useful tool!

Thanks a bunch for steering me towards it! :D
Bo Berglund
Sweden

hoserr9
Posts: 12
Joined: Mon Aug 24, 2020 5:05 pm

Re: Need to create Pi image to send to user

Fri Sep 18, 2020 9:05 pm

Hi Bosse_b
You mentioned that the size of the SD card was 32gb originally. When you create the image where you able to put it on say a 16Gb card since the actually contents of the RPI OS was only ~8GB.

Thanks
Jose

Bosse_B
Posts: 1074
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to create Pi image to send to user

Fri Sep 18, 2020 10:40 pm

hoserr9 wrote:
Fri Sep 18, 2020 9:05 pm
Hi Bosse_b
You mentioned that the size of the SD card was 32gb originally. When you create the image where you able to put it on say a 16Gb card since the actually contents of the RPI OS was only ~8GB.
Yes, the image is actual data space plus a little extra work space. But the 32 GB original imaged to less than 10 GB when done.
And I could gzip it to about half that in order to send it.
When one starts on that image in a new system you just use the raspi-config command to expand the file system to get to use whatever size your target SDcard is, no problem.
Bo Berglund
Sweden

hoserr9
Posts: 12
Joined: Mon Aug 24, 2020 5:05 pm

Re: Need to create Pi image to send to user

Fri Oct 09, 2020 7:31 am

I just want to note that IF you need to modify anything in the image you should mount the image like so:

https://askubuntu.com/questions/445979/ ... 942#538942

Code: Select all

$ fdisk -u -l rpi_image20201009.img

Disk rpi_image280914: 16.0 GB, 16012804096 bytes
255 heads, 63 sectors/track, 1946 cylinders, total 31275008 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cdac7

           Device Boot      Start         End      Blocks   Id  System
rpi_image20201009.img1   *        2048      514047      256000    c  W95 FAT32 (LBA)
rpi_image20201009.img2          540672    31242239    15350784   83  Linux
Then you must mount the second partition. Find the offset for the second partition Multiply 540672 * 512 = 276824064

Code: Select all

sudo mount -o loop,offset=276824064  /mnt
After you are done modifying the files unmount

Code: Select all

sudo umount  /mnt

hoserr9
Posts: 12
Joined: Mon Aug 24, 2020 5:05 pm

Re: Need to create Pi image to send to user

Fri Oct 09, 2020 8:09 am

Hi Bosse_b
[/quote]
When one starts on that image in a new system you just use the raspi-config command to expand the file system to get to use whatever size your target SDcard is, no problem.
[/quote]
Do you have the menu items in raspi-config to expand the file system?
If you take the image from a 32gb microSD card will the image expect 32gb?

--
Thanks
h

Return to “Raspberry Pi OS”