sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 11:57 pm

Liam Fraser said:



Download a 7zip file for that distro with the original image, with patch files for all different sizes
Verify the download with a hash
Extract the 7zip file


You do know that 7zip also verifies its archives?

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 12:20 am

Niel.

Thanks for the explaination on gunzip on the fly I was hoping but never got around to looking into that, so for this iso i am attemping to build in debian (being held back partly due to lack of support of wifi connection during install).

So attempting to run emulator sigh didn't want to mess with mounting vm's which will still require native linux to play with anyway.

I am using a netbook and some proggy just informed me that this version of processor (which i do not believe) Intel ATOM is amd64?

So looking into a nice distro to install native on the specially formatted 30Gig ext2 partition that is created in an extended logical partition #6 on sda.

Any ideas I am not a fan of Ubuntu, however the finished cd needs to be small to save on amount of downloading needed but still be supportive of the wifi and to flash the card for Pi.

Wasn't able to reply eariler because i was out costing a job for a new client.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 12:45 am

Also Neil.

How does mounting work during boot process?

I know it has to mount file system rw and even the hdd in grub at least it has root =somelongstringofrandomtext

and sometimes when you run command prompt (repair mode etc) it has to remount the file system..

If it is possible to mount root filesystem with resize on first boot then modify the initrd after this has completed (as linux can update initrd) or can replace it from a backup.

It shouldn't need a large script or too much messing around identifying its status as not being expanded and running into checksum errors later.

User avatar
jojopi
Posts: 3354
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 12:57 am

Neil Mayhew said:


Most of what I"ve read elsewhere says this won"t work. If the kernel is using the file system, even read-only, it"s not safe to change the partition table, and certainly not safe to resize the file system. For example, the resizing utility itself could get moved as it is running, and when the kernel needs to page some part of it into RAM, garbage will get loaded instead of the original code.


It is always[+] safe to increase the length of a partition without changing its start sector. No data have been written or moved; every block is exactly where it already was. The new blocks at the end, admittedly garbage, cannot be accessed until the filesystem itself is resized to fill them.

It is doubly safe in Linux because ioctl BLKRRPART fails when the filesystem is mounted. So actually neither the partition start nor length changes until the next reboot. The kernel remembers the original values just as if the partition table had not yet been changed at all.

Only the resize2fs stage actually adds or moves any data within the filesystem, and that is safe because it specifically checks that the kernel supports online resizing first.

([+]Okay, Linux md with superblock format 0.9 or 1.0 stores the raid metadata at the end of the physical partition, so extending these naively will cause them to be unrecognised on next boot. But even this is not insurmountable. And we can certainly assume that the Pi boot medium will not be raided.)


This is why a reboot is forced if repairs are done to the root file system during startup. Although fsck isn"t run from the initrd, the root fs is still read-only at that point, and a repair is typically way less drastic than a resize. For example, it"s unlikely that fsck or it"s dependent .so"s will get moved around or changed.


That is quite different. If you write to the block device underlying a mounted filesystem (as fsck does) then the in-memory and on-disk representations of the filesystem may be no longer in sync. If the filesystem is readonly then it would suffice to unmount and remount the filesystem in order to throw away the in-memory representation, but the root filesystem cannot be unmounted without rebooting.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 3:08 am

marc said:


I am using a netbook and some proggy just informed me that this version of processor (which i do not believe) Intel ATOM is amd64?


I don't know about all Atom's, but my D510 is definitely 64 bit.  (I'm running 64bit OS on it.)  Of course, you can run a 32 bit OS.  That's a significant reason why amd64 succeeded and the Itanium flopped.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 3:14 am

marc said:


How does mounting work during boot process?


It depends.


I know it has to mount file system rw and even the hdd in grub at least it has root =somelongstringofrandomtext


probably root=UUID={some UUID}

If it is possible to mount root filesystem with resize on first boot then modify the initrd after this has completed (as linux can update initrd) or can replace it from a backup.
It shouldn't need a large script or too much messing around identifying its status as not being expanded and running into checksum errors later.


I wouldn't bother replacing the initrd.  In fact, I don't see any reason for an initrd on the R-Pi.  But even if there is an initrd, no matter.  I wouldn't make it self-modifying.  It's easy enough for a script to detect if the partition fills the disk (if not, resize it then reboot) and if the filesystem fills the partition (if not, remount with resize to resize it then reboot).

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 7:31 am

HB said:


If you want to see a proof of concept now, the Ubuntu images for the PandaBoard do this.


Sounds promising


Unzipping and writing on the fly can be done easily enough with basic Unix utilities... "gunzip -c image_name.img | dd of=/dev/whatever" (example using gzip as I'm not familiar with 7zip's command line options off the top of my head). Doing this in Windows might be a bit tricky.


I think I'll be using the 7zip sdk to extract. I don't know how to go about doing that in Windows but I'd rather not rely on bash to do any part like that... need it to work nicely for everyone

I'll check if the 7zip verification is in the SDK too.

DDing in blocks should work fine too.

User avatar
jojopi
Posts: 3354
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 9:26 am

sylvan said:

I wouldn't bother replacing the initrd.  In fact, I don't see any reason for an initrd on the R-Pi.
Also, what we have been told about the boot sequence on the Pi is that the GPU loads kernel.img and cmdline.txt from FAT before booting the ARM -- no mention of initrd.  So I suspect that early distro releases will not support initrd until/unless a full bootloader is ported.

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 10:53 pm

I didn't realise so-called "online resizing" of mounted ext{2,3,4} filesystems was possible, and that certainly changes things. However, the small amount of research I've done suggests this isn't always the best solution, for two reasons:


Online resizing is less capable than offline (eg it doesn't increase the number of inodes, and doesn't play nicely with the flex_bg attribute)
When using regular partitions (ie not LVM or software RAID) a reboot is required before the increased partition size is recognized by the kernel and the filesystem can be resized to fill the extra space, so the main benefit of online resizing is lost

In addition, I've been looking into how Ubuntu does this for the PandaBoard and other OMAP systems, and they use modifications to the initrd. So I think I can't have been too far wide of the mark to suggest that in the first place.

Ubuntu uses a package called jasper. This actually does a bigger job than just resizing, because it customizes the installation for various things related to running on OMAP systems and using flash for the root fs. However, the resizing is done by installing a hook script and a couple of init scripts for initramfs-tools, which together put the necessary binaries into the initrd (eg fsck, resize2fs, sfdisk) and then do the resizing at boot time when it's needed.

As far as I can see, Ubuntu has done an excellent job in this area, and the best solution for the R? would be to port the jasper package to Debian, add support for anything R?-specific, and install the package into the official images.

Once this is done, the only use for a custom utility would be to smooth the process of downloading an official image, decompressing it and writing it to a card. Suitable card-writing utilities already exist for Windows and Linux (and, modulo a port, for Mac also), as do decompressors, so it's mainly a case of tying all those together in a nice way.

Others have pointed out that R? doesn't need an initrd, but most distros are set up to use one and I don't think R? loses anything by having one. I assume, for example, that the FAT partition will be mounted on /boot, so the normal kernel install process (which generates an initrd post-install) will "just work".

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 10:58 pm

This sounds like the ideal solution . However I think this will probably take a while to achieve and is certianly outside my knowledge range. I'll code what I plan and then can just remove the unnecessary features as the images themselves become more capable. Remember that is't not just Debian... also fedora and arch linux... XMBC and more. It's going to be a big job to build custom versions of these images. What is the way of knowing if something should be resized?

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:12 pm

Nice long explaination of just what i was looking for.

Online resizing for the rpi is an idea to help those who dont a. know how to do it. b. dont have a very large hdd or space left.

so a restart on the pi is not a big issue if its going to work without having to expand fs to partition on pc mac using chroot.

not sure but root for embedded devices might help such as busybox.

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:14 pm

Liam Fraser said:


Remember that is't not just Debian... also fedora and arch linux... XMBC and more.


Yes, good point. I didn't think of that until after I posted.


It's going to be a big job to build custom versions of these images.


I think they're all going to be R?-specific anyway, so adding the resizing logic isn't such a big deal. However, it does need to be supported by the teams developing the official R? images, so we need to communicate that to them.


What is the way of knowing if something should be resized?


jasper makes some changes to the root after it's run once, and it assumes that the fs always needs to be resized if these changes aren't already made. However, it tells sfdisk to grow the appropriate partition "as large as possible" and then tells resize2fs to grow the fs to fill the partition. If the fs already matches the partition, resize2fs does nothing to it.

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:20 pm

marc said:

Online resizing for the rpi is an idea to help those who dont a. know how to do it. b. dont have a very large hdd or space left.
so a restart on the pi is not a big issue if its going to work without having to expand fs to partition on pc mac using chroot.


The expansion will happen on the R? itself, not on the system doing the downloading and card writing. The original question was whether the resizing should be done while the boot process is still inside the initrd or later, after the kernel has switched to the real root fs.


not sure but root for embedded devices might help such as busybox.


The necessary code is all written and working in jasper, and is very portable to other distros. I think the shell used inside the initrd is already busybox, and all the scripts are executed with it.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:23 pm

The Rpi uses opensource software save for its rom.

Modifying boot processors is legal as long as you don"t mess with prop.

I reallythink having a check on each boot and installing vast amounts of software that executes on each boot is a waste of hardware resources.

livecd with the software to extract write on the fly.
sounds like a good plan or at least an option.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:23 pm

The Rpi uses opensource software save for its rom.

Modifying boot processors is legal as long as you don"t mess with prop.

I reallythink having a check on each boot and installing vast amounts of software that executes on each boot is a waste of hardware resources.

livecd with the software to extract write on the fly.
sounds like a good plan or at least an option.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:25 pm

Sounds like you know your stuff Neil!

Yeah as you say, eventually this will hopefully be adopted by the official Raspberry Pi images. However, it's way to far into the release now for them to change their images. As far as I know the image makers are still putting some final touches on the images but that's it. I think once we get this working and can show it to them... it has a good chance of being adopted. This would especially be effective if it was adopted for the educational release later in the year.

I'm off to bed now... college in the morning and exams this week!

Cheers,

Liam.

User avatar
Mezo
Posts: 55
Joined: Sun Jan 01, 2012 9:35 pm

Re: PiCard - GUI SD Preparation Tool

Tue Jan 10, 2012 11:43 pm

Hey guys, finished reading this thread late last night (every word) and just like to say thanks to everyone who`s working hard on this project (especially Liam).

I had been watching your tutorials Liam, then was reading this post as i was interested in disto`s & buying the correct SD card in preparation, its then i recognized your name in one of the posts.

I am (sad to say) your target audience, i want the Pi so i can learn Linux in a safe way without screwing my windys/microshite machine up, i still need it running sweet to admin my own forum etc.

I run everything on a teeny weeny little netbook with almost no hard disk space,i have no choice as my mains powered desktop is packed away in storage right now (no electricity in a caravan in a field) serious. So a utility to download an image on to a USB stick & then flash it direct to an SD card would be very handy indeed.

Would i be able then to us that program on the SD card using VirtualBox? i have an SD card slot on my netbook & it would be fantastic if i could tweak & play with it whilst im waiting for the Pi`s release & then simply slot the card in to the Pi & boot away.

Also,,,i want to learn a bit of programming as well for use with the Gertboard so i can control switches, relays, motors etc (electrician by trade) witch single language would you guys recommend for achieving this? remember im a complete novice at code, Python maybe?

Keep up the good work Liam & concentrate on the important things right now (study) as we all know good A-level results will help you at Uni.

Regards,

Mezo.

User avatar
jojopi
Posts: 3354
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 12:20 am

Neil Mayhew said:

Online resizing is less capable than offline (eg it doesn't increase the number of inodes,
resize2fs on a mounted filesystem does increase the number of inodes, to exactly the same extent that offline resizing would.

(Kernel-only (mount option) resizing does not add groups to the filesystem, so it can not increase the number of inodes, but nor can it increase the filesystem size beyond the configured bytes-per-inode ratio.)

and doesn't play nicely with the flex_bg attribute)
https://bugs.launchpad.net/ubuntu/+sour ... comments/1

Even Theodore Ts'o is not very concerned.

When using regular partitions (ie not LVM or software RAID) a reboot is required before the increased partition size is recognized by the kernel and the filesystem can be resized to fill the extra space,
Agreed.

so the main benefit of online resizing is lost
Only if you consider the main benefit of online resizing as not requiring a reboot. But I consider the main benefit of online resizing as not requiring an initrd. Plus an initrd always requires a reboot (compared to downloading a script and running it immediately).


As far as I can see, Ubuntu has done an excellent job in this area, and the best solution for the R? would be to port the jasper package to Debian, add support for anything R?-specific, and install the package into the official images.

l process (which generates an initrd post-install) will "just work".
I would agree, except that as I recently posted, I doubt that the R-Pi supports initrd booting at all. So at the very least jasper would need to be ported to run in the outside world.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 7:56 am

Mezo said:


I run everything on a teeny weeny little netbook with almost no hard disk space,i have no choice as my mains powered desktop is packed away in storage right now (no electricity in a caravan in a field) serious. So a utility to download an image on to a USB stick & then flash it direct to an SD card would be very handy indeed.


Out of curiosity Mezo, how much free space do you have to work with? So I have an idea of a typical end users free space. If not then yeah USB stick might be ideal to use as external storage



Would i be able then to us that program on the SD card using VirtualBox? i have an SD card slot on my netbook & it would be fantastic if i could tweak & play with it whilst im waiting for the Pi`s release & then simply slot the card in to the Pi & boot away.


This wont be possible because the Raspberry Pi images are very specific to the Pi. They will only boot on the Pi and also are ARM archetecture which won't be supported by VirtualBox. I know it would be ideal !



Also,,,i want to learn a bit of programming as well for use with the Gertboard so i can control switches, relays, motors etc (electrician by trade) witch single language would you guys recommend for achieving this? remember im a complete novice at code, Python maybe?


By the looks of the video that gert put up... you can use the drivers from whatever language you want as you just have to write some data to a certian place in Linux. I'd say python because it's easy to learn and it will be really easy to use the gertboard drivers with it.




Thanks for your kind words

Cheers,

Liam.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 8:10 am

Mornin Liam,

Have a good day today hehe.

I removed the linux partition off my netbook without restoring my mbr lol.

Had a bit of fun with grub boot-loader and customizing grub.cfg to boot windows /dev/sda3 partition. sigh sorted out now, used F5 - F8 option to get command line and fix mbr and boot.

Was looking into a very small linux distro that might be useful it runs out of dos and loads a small ram drive of around 256MB, which customized can be used to run your program, it comes fully loaded with all linux commands and most apps used in command line such as fdisk partition manager etc. Best of it is that it could be less than 20Mb Download!

Got it running on vm atm,, can copy all files from the virtual hdd to usb using the usb input on virtualbox.  Then its a simple case to ms-sys an iso.

What do you think?

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 8:57 am

Hi Marc. It sounds good as long as it has drivers for usb memory card readers and all that. It will have to run Java and a window manager of course.

It will still have to be distributed as a fairly large iso because theres no way I can extract an image in 256mb of ramdisk. That means you'd have to have one livecd for each distro which is a little bit more work. That could be automated though just like my back end will be.

To be honest, at the moment I need to just get the utility working in standard environments. I don't have the time to get it working in a live cd. Of course you are more than welcome to see how the utility performs in a livecd environment once I release it initially.

I'm at college now anyway so wont be able to reply till tonight.

Cheers,

Liam.

User avatar
Mezo
Posts: 55
Joined: Sun Jan 01, 2012 9:35 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 9:20 am

It has a 14 gig solid state HD with 3 gigs space left, which is not good as they say you should only fill half of the drive up for best performance, but you know how windys bloats itself eating all them pies over the years (even with system restore switched off from day one).

Cheers, Mezo.

Liam Fraser said:

Out of curiosity Mezo, how much free space do you have to work with? So I have an idea of a typical end users free space. If not then yeah USB stick might be ideal to use as external storage

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 6:18 pm

It all depends on how large the downloads are I don't have any distro images yet. Realistically though I highly doubt 3gb will be enough. Whatever is downloaded will then need to be extracted so yeah a USB drive would probably be better. Out of curiosity how large is your USB drive?

User avatar
grumpyoldgit
Posts: 1452
Joined: Thu Jan 05, 2012 12:20 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 6:38 pm

Liam Fraser said:


It all depends on how large the downloads are I don't have any distro images yet. Realistically though I highly doubt 3gb will be enough. Whatever is downloaded will then need to be extracted so yeah a USB drive would probably be better. Out of curiosity how large is your USB drive?



Liam Fraser said:


It all depends on how large the downloads are I don't have any distro images yet. Realistically though I highly doubt 3gb will be enough. Whatever is downloaded will then need to be extracted so yeah a USB drive would probably be better. Out of curiosity how large is your USB drive?



As a guide, the Ubuntu distro fits on a CD and I would assume the other Linux distros would be similar. What that expands out to is another matter. My boot drive is now using just over 6Gb but I have been installing all sorts of stuff such as LibreOffice, Banshee, Wine etc,  and I have been through several upgrades so there must be all sorts of junk. All data, however, is filed externally.  It would be interesting to know what a minimal Linux installation uses.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 11, 2012 7:02 pm

I suppose it really depends on what distro they are using and how light it is... although they'll all be pretty light compared to what bloaty stuff we see nowadays !

Return to “Other projects”