slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

pi-gen : no space left on device

Tue Jan 21, 2020 12:43 pm

Hi all.

I've had a few issues trying to run pi-gen on an x86 AMD processor with Ubuntu and also with Debian.

I eventually gave up and spun up an ARM Ubuntu machine on amazon aws.

This was much better.

I ran the pi-gen script fresh after git check out. The first 2 images built with no error.

While building the full image I got the following error just after the script attempts to Get:2 http://raspbian.raspberrypi.org/raspbian buster InRelease

Error writing to file - write (28: No space left on device)

My suspicions are that the defined image file is too small. Anyone know where in the pi-gen code I can increase it?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 1:00 pm

pi-gen does all the work on the existing filesystem. At the end, it will create an image. The image size will depend on the size of what goes in it. So no, I don't think that's the issue.

pi-gen is a bit dumb and will just copy every stage across. That means you end up with about 7 copies of the files, which can easily eat up the space provided on an aws instance.

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 1:18 pm

At the end, it will create an image. The image size will depend on the size of what goes in it.
Thank you - I was wondering about this.

I just looked at my df - h on my amazon instance.

I have 51G left, which is 51% left of the disk.
So lots of space there.

Any further ideas?

Code: Select all

Filesystem       Size  Used Avail Use% Mounted on
udev             1.9G     0  1.9G   0% /dev
tmpfs            381M  888K  380M   1% /run
/dev/nvme0n1p1    97G   47G   51G  49% /
tmpfs            1.9G     0  1.9G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/nvme0n1p15   98M  122K   98M   1% /boot/efi
/dev/loop0        79M   79M     0 100% /snap/core/7399
/dev/loop1        16M   16M     0 100% /snap/amazon-ssm-agent/1454
tmpfs            381M     0  381M   0% /run/user/1000
You can see the /dev/loop have 0 space available, could these be the loops created and used by pi-gen - they look far too small though.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 2:00 pm

The loopback devices are used by snaps - canonical's take on distribution-independent packages.

What's the other output from pi-gen before the error?

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 3:27 pm

Yes, at this point pi-gen has managed to build a lite version.
pi-gen then starts to try and generate a full image, i guess that's the beginning 00-run.sh
I've highlighted the errors in bold.

Thank you for replying so quickly. :)

Code: Select all

Stopping fake hwclock: saving system time.
Mode:     real
Files:    5744
Linked:   1386 files
Compared: 0 xattrs
Compared: 3082 files
Saved:    44.50 MiB
Duration: 0.52 seconds
  adding: 2020-01-21-MYIMAGE.img (deflated 69%)
[14:00:31] End /home/ubuntu/pi-gen/pi-gen/export-image/04-finalise/01-run.sh
[14:00:31] End /home/ubuntu/pi-gen/pi-gen/export-image/04-finalise
[14:00:31] End /home/ubuntu/pi-gen/pi-gen/export-image
[14:00:31] Begin /home/ubuntu/pi-gen/pi-gen/export-noobs
[14:00:31] Begin /home/ubuntu/pi-gen/pi-gen/export-noobs/prerun.sh
/boot: offset 4194304, length 268435456
/:     offset 272629760, length 3586129920
'/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/rootfs/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service' -> '/lib/systemd/system/apply_noobs_os_config.service'
[14:19:32] End /home/ubuntu/pi-gen/pi-gen/export-noobs/prerun.sh
[14:19:32] Begin /home/ubuntu/pi-gen/pi-gen/export-noobs/00-release
[14:19:32] Begin /home/ubuntu/pi-gen/pi-gen/export-noobs/00-release/00-run.sh
'files/partition_setup.sh' -> '/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/2020-01-21-MYIMAGE/partition_setup.sh'
'files/partitions.json' -> '/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/2020-01-21-MYIMAGE/partitions.json'
'files/os.json' -> '/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/2020-01-21-MYIMAGE/os.json'
'files/OS.png' -> '/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/2020-01-21-MYIMAGE/OS.png'
'files/release_notes.txt' -> '/home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-noobs/2020-01-21-MYIMAGE/release_notes.txt'
./
./slides_vga/
./slides_vga/B.png
./slides_vga/E.png
./slides_vga/A.png
./slides_vga/F.png
./slides_vga/D.png
./slides_vga/G.png
./slides_vga/C.png

[14:19:48] End /home/ubuntu/pi-gen/pi-gen/export-noobs/00-release/00-run.sh
[14:19:48] End /home/ubuntu/pi-gen/pi-gen/export-noobs/00-release
[14:19:48] End /home/ubuntu/pi-gen/pi-gen/export-noobs
[14:19:48] Begin /home/ubuntu/pi-gen/pi-gen/export-image
[14:19:48] Begin /home/ubuntu/pi-gen/pi-gen/export-image/prerun.sh
/boot: offset 4194304, length 268435456
/:     offset 272629760, length 6564085760
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.44.1 (24-Mar-2018)
mount: /dev/loop5 mounted on /home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-image/rootfs.
mount: /dev/loop4 mounted on /home/ubuntu/pi-gen/pi-gen/work/2020-01-21-MYIMAGE/export-image/rootfs/boot.
[14:28:39] End /home/ubuntu/pi-gen/pi-gen/export-image/prerun.sh
[14:28:39] Begin /home/ubuntu/pi-gen/pi-gen/export-image/00-allow-rerun
[14:28:39] Begin /home/ubuntu/pi-gen/pi-gen/export-image/00-allow-rerun/00-run.sh
[14:28:39] End /home/ubuntu/pi-gen/pi-gen/export-image/00-allow-rerun/00-run.sh
[14:28:39] End /home/ubuntu/pi-gen/pi-gen/export-image/00-allow-rerun
[14:28:39] Begin /home/ubuntu/pi-gen/pi-gen/export-image/01-set-sources
[14:28:39] Begin /home/ubuntu/pi-gen/pi-gen/export-image/01-set-sources/01-run.sh
Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Err:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
  Error writing to the file - write (28: No space left on device) [IP: XXX.XXX.XXX.XXX]
Hit:2 http://archive.raspberrypi.org/debian buster InRelease
Err:2 http://archive.raspberrypi.org/debian buster InRelease
  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.raspberrypi.org/debian buster InRelease: Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease  Error writing to the file - write (28: No space left on device) [IP: XXX.XXX.XXX.XXX]
W: Failed to fetch http://archive.raspberrypi.org/debian/dists/buster/InRelease  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[14:28:44] End /home/ubuntu/pi-gen/pi-gen/export-image/01-set-sources/01-run.sh

Err:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Error writing to the file - write (28: No space left on device) [IP:XXX.XXX.XXX.XXX]
Hit:2 http://archive.raspberrypi.org/debian buster InRelease
Err:2 http://archive.raspberrypi.org/debian buster InRelease
Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.raspberrypi.org/debian buster InRelease: Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://raspbian.raspberrypi.org/raspbia ... /InRelease Error writing to the file - write (28: No space left on device) [IP: XXX.XXX.XXX.XXX]
W: Failed to fetch http://archive.raspberrypi.org/debian/d ... /InRelease Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Some index files failed to download. They have been ignored, or old ones used instead.
Last edited by slice-o-pie on Tue Jan 21, 2020 3:36 pm, edited 1 time in total.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 3:34 pm

What's the underlying filesystem / is on? Is it ext4 or something else?

trejan
Posts: 1112
Joined: Tue Jul 02, 2019 2:28 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 3:41 pm

Ran out of inodes? Run "df -i"

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 3:44 pm

looks like ext4.

Code: Select all

>df-Th

Filesystem      Type      Size  Used Avail Use% Mounted on
udev            devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs           tmpfs     381M  880K  380M   1% /run
/dev/nvme0n1p1  ext4       97G   47G   51G  49% /
tmpfs           tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs           tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs           tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/loop0      squashfs   79M   79M     0 100% /snap/core/7399
/dev/loop1      squashfs   16M   16M     0 100% /snap/amazon-ssm-agent/1454
/dev/nvme0n1p15 vfat       98M  122K   98M   1% /boot/efi
/dev/loop2      squashfs   79M   79M     0 100% /snap/core/8271
/dev/loop3      squashfs   16M   16M     0 100% /snap/amazon-ssm-agent/1479
tmpfs           tmpfs     381M     0  381M   0% /run/user/1000

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 3:45 pm

Code: Select all

[email protected]:~$ df -i
Filesystem        Inodes  IUsed    IFree IUse% Mounted on
udev              483499    357   483142    1% /dev
tmpfs             486419    540   485879    1% /run
/dev/nvme0n1p1  12966400 515528 12450872    4% /
tmpfs             486419      1   486418    1% /dev/shm
tmpfs             486419      3   486416    1% /run/lock
tmpfs             486419     18   486401    1% /sys/fs/cgroup
/dev/loop0         12803  12803        0  100% /snap/core/7399
/dev/loop1            15     15        0  100% /snap/amazon-ssm-agent/1454
/dev/nvme0n1p15        0      0        0     - /boot/efi
/dev/loop2         12822  12822        0  100% /snap/core/8271
/dev/loop3            15     15        0  100% /snap/amazon-ssm-agent/1479
tmpfs             486419     11   486408    1% /run/user/1000

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 4:12 pm

I've also read you can use pi-gen on raspian on a pi4?
I'll give that a try also.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 6:01 pm

Not sure what the issue is then.

If you suspect the image size may be an issue, putting a 'df' into export-image/01-set-sources/01-run.sh at a point before it fails could confirm that.

Yes, you can use pi-gen on a pi, but make sure you have plenty of storage.

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 6:27 pm

That inode issue... i had a 100GB magnetic disk on amazon. Shouldn't be a problem.
I decided to try using an SSD with my amazon image instead.

Interestingly, it fails at the same point but with no memory error! odd.

into data and signature failed

Looks like a signature issue. Any idea what to test next?

Code: Select all

[17:06:46] Begin /home/ubuntu/pi-gen/export-image/prerun.sh
/boot: offset 4194304, length 268435456
/:     offset 272629760, length 6564085760
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.44.1 (24-Mar-2018)
mount: /dev/loop3 mounted on /home/ubuntu/pi-gen/work/2020-01-21-MY_IMAGE/export-image/rootfs.
mount: /dev/loop2 mounted on /home/ubuntu/pi-gen/work/2020-01-21-MY_IMAGE/export-image/rootfs/boot.
[17:09:31] End /home/ubuntu/pi-gen/export-image/prerun.sh
[17:09:31] Begin /home/ubuntu/pi-gen/export-image/00-allow-rerun
[17:09:31] Begin /home/ubuntu/pi-gen/export-image/00-allow-rerun/00-run.sh
[17:09:31] End /home/ubuntu/pi-gen/export-image/00-allow-rerun/00-run.sh
[17:09:31] End /home/ubuntu/pi-gen/export-image/00-allow-rerun
[17:09:31] Begin /home/ubuntu/pi-gen/export-image/01-set-sources
[17:09:31] Begin /home/ubuntu/pi-gen/export-image/01-set-sources/01-run.sh
Hit:1 http://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Err:1 http://archive.raspberrypi.org/debian buster InRelease
  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
Err:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
  Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.raspberrypi.org/debian buster InRelease: Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://raspbian.raspberrypi.org/raspbian buster InRelease: Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease  Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://archive.raspberrypi.org/debian/dists/buster/InRelease  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[17:09:40] End /home/ubuntu/pi-gen/export-image/01-set-sources/01-run.sh
[17:09:40] End /home/ubuntu/pi-gen/export-image/01-set-sources
[17:09:40] Begin /home/ubuntu/pi-gen/export-image/02-network
[17:09:40] Begin /home/ubuntu/pi-gen/export-image/02-network/01-run.sh
[17:09:40] End /home/ubuntu/pi-gen/export-image/02-network/01-run.sh
[17:09:40] End /home/ubuntu/pi-gen/export-image/02-network

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 8:52 pm

Well.... looks like your initial assessment was correct:

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      5.9G  5.7G     0 100% /
/dev/loop0      253M   53M  200M  21% /boot
udev            7.8G     0  7.8G   0% /dev
Hit:1 http://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Err:1 http://archive.raspberrypi.org/debian buster InRelease
  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
Err:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
  Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.raspberrypi.org/debian buster InRelease: Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://raspbian.raspberrypi.org/raspbian buster InRelease: Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease  Splitting up /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_buster_InRelease into data and signature failed
W: Failed to fetch http://archive.raspberrypi.org/debian/dists/buster/InRelease  Splitting up /var/lib/apt/lists/archive.raspberrypi.org_debian_dists_buster_InRelease into data and signature failed
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
We need to come up with a better way to calculate the image size to create, because it doesn't scale properly.

IIRC, right now it's the size of the files + some padding. Maybe it needs to be some multiple of the size of the files, like 1.2.

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 9:56 pm

IIRC, right now it's the size of the files + some padding. Maybe it needs to be some multiple of the size of the files, like 1.2.
Do you know which parts of the pi-gen code I can hit with a hammer? :)

I'd like to increase the size of the image manually. :)

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Tue Jan 21, 2020 10:27 pm

It's the mess in export-image prerun script.

Trying this out right now:

Code: Select all

diff --git a/export-image/prerun.sh b/export-image/prerun.sh
index cecde32..8a4b935 100755
--- a/export-image/prerun.sh
+++ b/export-image/prerun.sh
@@ -18,7 +18,7 @@ ALIGN="$((4 * 1024 * 1024))"
 # some overhead (since actual space usage is usually rounded up to the
 # filesystem block size) and gives some free space on the resulting
 # image.
-ROOT_MARGIN=$((800*1024*1024))
+ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2) / 1" | bc)"
 
 BOOT_PART_START=$((ALIGN))
 BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN))

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Tue Jan 21, 2020 10:44 pm

Yes. I just spent a glass of wine tracking down line 13.

https://github.com/RPi-Distro/pi-gen/bl ... /prerun.sh

Thank you for running it. 😊

slice-o-pie
Posts: 10
Joined: Tue Jan 21, 2020 12:33 pm

Re: pi-gen : no space left on device

Fri Jan 24, 2020 7:17 pm

@shiftplusone

Your change made it work! Thank you!

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6177
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: pi-gen : no space left on device

Fri Jan 24, 2020 8:27 pm

Thanks for reporting the issue.

Return to “Advanced users”