flyinginclouds
Posts: 5
Joined: Mon Feb 13, 2017 9:14 pm

Automatic Expand Filesystem after cloning a Rasbian image

Mon Feb 13, 2017 9:39 pm

I installed raspbian Jessie on SD card (8GB) and made some changes to the OS files for my personal project. After cloning the SD card (used Win32DiskImager for cloning) , I installed the OS image on an 32 GB sd card. I had to manually expand the file system (via Raspi-config) to use full 32 GB sd card.

Is there a way so that cloned image automatically expands the file system on boot up without any manual work ?

klricks
Posts: 6706
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA
Contact: Website

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 14, 2017 12:35 am

flyinginclouds wrote:I installed raspbian Jessie on SD card (8GB) and made some changes to the OS files for my personal project. After cloning the SD card (used Win32DiskImager for cloning) , I installed the OS image on an 32 GB sd card. I had to manually expand the file system (via Raspi-config) to use full 32 GB sd card.

Is there a way so that cloned image automatically expands the file system on boot up without any manual work ?
You would need to create a script of your own to do that on first boot then the script would need to disable or delete itself.
You can look at raspi-config to see how expand is accomplished and base your script on that:

Code: Select all

nano /usr/bin/raspi-config
Unless specified otherwise my response is based on the latest and fully updated Raspbian Buster w/ Desktop OS.

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 14, 2017 11:56 am

If you want to imitate the way Raspbian expands the filesystem on first boot, here is how it works:

First edit /boot/config.txt and append init=/usr/lib/raspi-config/init_resize.sh

Then create the file /etc/init.d/resize2fs_once containing the following (copied from raspi-config) and enable it with sudo chmod +x /etc/init.d/resize2fs_once and sudo systemctl enable resize2fs_once

Code: Select all

#!/bin/sh
### BEGIN INIT INFO
# Provides:          resize2fs_once
# Required-Start:
# Required-Stop:
# Default-Start: 3
# Default-Stop:
# Short-Description: Resize the root filesystem to fill partition
# Description:
### END INIT INFO
. /lib/lsb/init-functions
case "\$1" in
  start)
    log_daemon_msg "Starting resize2fs_once" &&
    resize2fs /dev/$ROOT_PART &&
    update-rc.d resize2fs_once remove &&
    rm /etc/init.d/resize2fs_once &&
    log_end_msg \$?
    ;;
  *)
    echo "Usage: \$0 start" >&2
    exit 3
    ;;
esac
(At this point you would make your cloned image)

Then on the next boot init_resize.sh will run, repartition the SD card, re-modify /boot/config.txt so that it is not run a second time, and reboot.

On the following boot resize2fs_once will run, expand the filesystem to fill the new partition, and then disable (and delete) itself.

flyinginclouds
Posts: 5
Joined: Mon Feb 13, 2017 9:14 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Mon Feb 20, 2017 5:02 am

Hello TimG

I tried your steps:
1. Appended init=/usr/lib/raspi-config/init_resize.sh to /boot/config.txt

2. Create the file /etc/init.d/resize2fs_once containing the following (copied from raspi-config)
http://imgur.com/a/QQoPr

3. Enable it with sudo chmod +x /etc/init.d/resize2fs_once and sudo systemctl enable resize2fs_once
http://imgur.com/a/ZzSRh

4. I cloned the image and the filesize never expanded. I checked the service status and it mentions that it failed.
http://imgur.com/a/ebvgU

Can you tell me if I am making any error ?

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Mon Feb 20, 2017 3:00 pm

My mistake. The version of resize2fs_once from raspi-config is slightly different, and depends on variables created by raspi-config. To be sure of getting the right version, download it directly from the RPi Github:

Code: Select all

sudo wget -O /etc/init.d/resize2fs_once https://github.com/RPi-Distro/pi-gen/raw/dev/stage2/01-sys-tweaks/files/resize2fs_once
sudo chmod +x /etc/init.d/resize2fs_once
sudo systemctl enable resize2fs_once
In case you want to test without going to the trouble of making an image and transferring it to another card:
  • Write the Raspbian image as normal
  • Before booting for the first time, edit cmdline.txt and delete the init=... section
  • Boot as normal and verify that the file system has not been expanded
  • Restore the init=... section to cmdline.txt
  • Run the 3 lines of code above
  • Reboot
  • Check that the filesystem has been properly expanded to fill the card
Aside:
This is the sequence I use to create a separate data partition on the SD-card (via modified versions of the init_resize.sh and resize2fs_once scripts). The partition containing the OS runs read-only to reduce the likelihood of accidental corruption, whilst the data partition is read-write. The data partition might still be corrupted if there is an unlucky power interruption, but that can be fixed automatically by a script so long as the root partition is intact.

flyinginclouds
Posts: 5
Joined: Mon Feb 13, 2017 9:14 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 21, 2017 5:55 am

TimG

Still unsuccessful. Here is what I did now.

1. Installed a cloned image (8GB) to a 32 GB SD Card.

2. Pi starts and size is 8GB.

3. Added init=/usr/lib/raspi-config/init_resize.sh to /boot/cmdline.txt
http://imgur.com/a/XxRhf

4. Copied contents from github repo to resize2fs_once
sudo chmod +x /etc/init.d/resize2fs_once
sudo update-rc.d resize2fs_once defaults ( I also tried sudo systemctl enable resize2fs_once, but results were similar)
http://imgur.com/a/KYZuO

5. Rebooted the pi and no change in disk size.

6. I checked the service status and see that it exited successfully and resize2fs_once is removed from /etc/init.d directory.
http://imgur.com/a/HcebM

7. I checked the daemon log and it has the message: http://imgur.com/a/ZwRc3 (Look at the last three lines)
Feb 21 05:35:07 raspberrypi resize2fs_once[447]: Starting resize2fs_once:resize2fs 1.42.12 (29-Aug-2014)
Feb 21 05:35:07 raspberrypi resize2fs_once[447]: The filesystem is already 1908736 (4k) blocks long. Nothing to do!
Feb 21 05:35:08 raspberrypi resize2fs_once[447]: .

8. I am positive I am using a 32 GB SD Card.
Last edited by flyinginclouds on Thu Feb 23, 2017 10:03 pm, edited 1 time in total.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6115
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 21, 2017 6:28 am

flyinginclouds wrote:init=/usr/lib/raspi-config/init_resize.s
try

Code: Select all

quiet init=/usr/lib/raspi-config/init_resize.sh

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 21, 2017 5:13 pm

Clouds,
Is there a line-break in your cmdline.txt? IIRC everything needs to be on one line.

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 21, 2017 10:59 pm

After a little digging I came up with a couple of lines to download the official cmdline.txt and apply the official patch to trigger the repartition script.

Code: Select all

sudo wget -qO /boot/cmdline.txt https://github.com/RPi-Distro/pi-gen/raw/dev/stage1/00-boot-files/files/cmdline.txt
wget -qO - https://github.com/RPi-Distro/pi-gen/raw/dev/stage2/01-sys-tweaks/00-patches/07-resize-init.diff | sudo patch -p3 -d /boot
Don't forget the script to resize the filesystem to fill the partition:

Code: Select all

sudo wget -qO /etc/init.d/resize2fs_once https://github.com/RPi-Distro/pi-gen/raw/dev/stage2/01-sys-tweaks/files/resize2fs_once
sudo chmod +x /etc/init.d/resize2fs_once
sudo systemctl enable resize2fs_once
Then reboot.

flyinginclouds
Posts: 5
Joined: Mon Feb 13, 2017 9:14 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Tue Feb 21, 2017 11:24 pm

Here is the update:

1. Added quiet init=/usr/lib/raspi-config/init_resize.sh to /boot/cmdline.txt.
It is not a newline. Added it in continuation of the previous line. This is important. This is the only change made wrt to previous attempt.
http://imgur.com/a/Y3LME

2. Copied contents from github repo to resize2fs_once
sudo chmod +x /etc/init.d/resize2fs_once
sudo update-rc.d resize2fs_once defaults
http://imgur.com/a/KYZuO

3. Made the cloned image (8GB in size)

4. Burned the cloned image on a 32 GB SD Card.

5. Power it up and the very first time, the Pi expands the filesystem and it works. resize2fs_once is rm after first boot and it doesnt run again.

6. cmdline.txt no longer has the quiet init=/usr/lib/raspi-config/init_resize.sh. Is this normal ?
http://imgur.com/a/J1MUS

Thanks TimG and ShiftPlusOne. :D :D :D :D
Last edited by flyinginclouds on Thu Feb 23, 2017 10:04 pm, edited 1 time in total.

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Wed Feb 22, 2017 12:11 am

flyinginclouds wrote:6. cmdline.txt no longer has the quiet init=/usr/lib/raspi-config/init_resize.sh. Is this normal ?
Yes - otherwise you'd get stuck in a reboot loop. Line 132 of init_resize.sh makes the change.


davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Mon Mar 06, 2017 12:38 pm

Is it necessary to remove the resize once daemon, (versus disabling)?

That way, next time I shrink the filesystem for uploading to a website where others can download, I need only the enable the daemon using

sudo systemctl enable resize2fs_once

And append the text to /boot/cmdline.txt using sed in something like /etc/rc.local or other script that runs last (or at least late) in the boot up process.

In other words, not have to do these steps every time I want to prepare a new, shrunk image for distribution.

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Mon Mar 06, 2017 11:40 pm

Just edit the resize2fs_once script and delete the line

Code: Select all

rm /etc/init.d/resize2fs_once &&
But be sure to leave the line

Code: Select all

update-rc.d resize2fs_once remove &&
Beware of editing cmdline.txt with an init script. You'll end up running init_resize.sh on every boot (followed by a shutdown and a reboot).


davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Wed Mar 08, 2017 8:52 pm

Beware of editing cmdline.txt with an init script. You'll end up running init_resize.sh on every boot (followed by a shutdown and a reboot).
I want to make sure I understand...

Everything works when I run the wgets and enable the daemon manually. But, I want the automatic expansion stuff enabled as part of an image cloning (and shrinking) script that can't have a dependency on a internet connection.

So, I edited /etc/init.d/resize2fs_once to eliminate the line that deletes itself. I wrote a simple script which enabled the daemon and used sed to append :

Code: Select all

quiet init=/usr/lib/raspi-config/init_resize.sh 
to /boot/cmdline.txt

As you pointed out earlier, init_resize.sh does a sed command to truncate cmdline.txt to make sure it only gets called once.

I don't run the script in a "init script". I just run it as part of a script that shrinks the file system.

For some reason, it didn't work correctly. I end up with a blinking cursor. But, it works everytime when I do it "manually".

There's probably something subtle with the text I appended with sed, but I wanted to make sure I'm not missing your point before trying again. This time, I'll save the script somewhere else before enabling the automatic expansion mechanisms!

Thanks!








to /boot/cmdline.txt

davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Wed Mar 08, 2017 9:09 pm

Actually, I'm thinking that the problem was that I used:

Code: Select all

sudo systemctl enable resize2fs_once
in my script to enable the resizing. But, maybe after doing

Code: Select all

update-rc.d resize2fs_once remove

versus

Code: Select all

systemctl disable  resize2fs_once
another update-rc.d command is needed to ADD the service versus enabling it?

I'm new to systemctl verus update-rc.d, so maybe I need to do my homework on that.

davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 6:03 pm

At least once everything worked when I did exactly as TimG suggested:

Code: Select all

sudo wget -qO /boot/cmdline.txt https://github.com/RPi-Distro/pi-gen/raw/dev/stage1/00-boot-files/files/cmdline.txt
wget -qO - https://github.com/RPi-Distro/pi-gen/raw/dev/stage2/01-sys-tweaks/00-patches/07-resize-init.diff | sudo patch -p3 -d /boot
sudo wget -qO /etc/init.d/resize2fs_once https://github.com/RPi-Distro/pi-gen/raw/dev/stage2/01-sys-tweaks/files/resize2fs_once
sudo chmod +x /etc/init.d/resize2fs_once
sudo systemctl enable resize2fs_once
But, when I've tried to do that on more clones of working Jessie Lite images, they won't boot. They hang with just the 4 raspberries, or a couple of lines like "No Caching mode page found" and "Assuming drive cache: write through". But, usually just the 4 berries.

The cloned images booted fine until I ran the above commands that patched cmdline.txt, and downloaded/patched and enabled the resize2fs_once daemon.

cmdline.txt and resize2fs_once look right before I do the reboot.

After failing to boot, I can put the image in a USB card reader and mount it. When I look at

/media/pi/boot/cmdline.txt

I see that the line hasn't been truncated, which indicates /usr/lib/raspi-config/init_resize.sh didn't run far enough to do that.

Similarly, /media/pi/xxxx/etc/init.d/resize_2fs_once still exists. Which, indicates it didn't run.

What can I do to debug?

Thanks!

Tried unsuccessfully three times. Is there a way to debug what's going wrong?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6115
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 6:15 pm

I don't think that patch to cmdline.txt will apply.

davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 9:36 pm

I don't think that patch to cmdline.txt will apply
Could you elaborate?

The patch appends the invocation of /usr/lib/raspi-config/init_resize.sh to cmdline.txt. That's what resizes the file system.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6115
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 10:04 pm

davethomaspilot wrote:
I don't think that patch to cmdline.txt will apply
Could you elaborate?

The patch appends the invocation of /usr/lib/raspi-config/init_resize.sh to cmdline.txt. That's what resizes the file system.
This commit would break the patch for your use case:
https://github.com/RPi-Distro/pi-gen/co ... 9c6adbdb07

davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 10:57 pm

Ok, thanks!

I do see a difference in the patch to cmdline.txt besides the appended txt.

I'll try appending the text for invoking the script to a known, working cmdline.txt.

davethomaspilot
Posts: 107
Joined: Tue Apr 29, 2014 6:18 pm

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 11:06 pm

HOORAY! It works!

Thank you very much!

When it worked, I must have been editing cmdline.txt manually versus applying the patch.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6115
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 11:10 pm

You may have been applying the patch before that commit was pushed to the repo, in which case I would also expect it to work.

Aside from that, it might be worth nothing that the latest version of the resize script (contained within the raspi-config package) is a bit less dumb. The quiet parameter is no longer required and the script works when booting off USB too.

User avatar
TimG
Posts: 295
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Automatic Expand Filesystem after cloning a Rasbian imag

Thu Mar 16, 2017 11:39 pm

For me the recipe still works with the updated official cmdline.txt patch file. It will fail if you forget the first line, which downloads the matching official cmdline.txt.

Return to “Advanced users”