sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 4:14 pm

Montekuri said:


And if this is a ISO img we can just use ISO to USB app to write to a SD card:

http://www.isotousb.com/

We can use a USB Card Reader to write to the SD card:

http://www.amazon.com/Mini-Mem.....038;sr=1-1


We'll definitely need the card reader.  But the iso to usb app would do 99.9% the wrong thing. An ISO is a PC bootable CD rom filesystem.  ISO to USB is designed to make a usb drive bootable on a PC.  That's just wrong.

khannie
Posts: 1
Joined: Wed Jan 25, 2012 4:47 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 4:53 pm

For linux users I think dd would work very well:

dd if=<image file> of=<sd card device>

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 4:57 pm

khannie said:


For linux users I think dd would work very well:

dd if=<image file> of=<sd card device>



except for one thing...   the size of the device is encoded in the partition table (typically the first sector) and if you overwrite that, things get confusing later.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:27 pm

The official SD card images will be distributed as a DD image file that can be written with dd or a similar imaging tool for windows.

The size of the actual device is not stored in the partition table... but the size of the partitions is. The user will need to either DD and then resize the partitions accordingly... or use this tool which will have a number of images per distro, for different card sizes all the way from 2gb up to 32gb

This will not mean downloading multiple images for different cards. Rather using binary difference algorithms to build patches for the original DD, which will patch the difference when partition size is increased to the original.

Hope this clears some things up,

Liam.

davidgoodenough
Posts: 74
Joined: Wed Sep 21, 2011 11:55 am

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:28 pm

sylvan said:


khannie said:


For linux users I think dd would work very well:

dd if=<image file> of=<sd card device>


except for one thing...   the size of the device is encoded in the partition table (typically the first sector) and if you overwrite that, things get confusing later.


Actually you can use any of the parted family (parted itself, or gparted or qtparted) to fix the partition table after you make the copy.  This only works if the device is bigger than the image, but it does work - I do it all the time.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:32 pm

Liam Fraser said:


The size of the actual device is not stored in the partition table...


You are wrong about that.

The table stores the size of the device and the size of each partition.  The table also stores the CHS geometry, but that is mostly irrelevant these days as everything uses LBA instead.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:37 pm

davidgoodenough said:


sylvan said:


except for one thing...   the size of the device is encoded in the partition table (typically the first sector) and if you overwrite that, things get confusing later.


Actually you can use any of the parted family (parted itself, or gparted or qtparted) to fix the partition table after you make the copy.  This only works if the device is bigger than the image, but it does work - I do it all the time.


Is that before or after the kernel reads the incorrect size from the partition table?

I will admit I mostly use parted only when I need GPT partitioning instead of MBR.  However I was doing this exact operation recently and parted got very confused if the kernel had read the incorrect data from the MBR.  The cleanest recovery I found was to wipe the MBR (dd if=/dev/zero of=/dev/sdg bs=512 count=1;  then either unplug/replug or sfdisk -R /dev/sdg) and create the partitions from scratch (not touching the data in the partition, of course).

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:46 pm

It doesnt affect the way this tool will work though. In terms of writing to disk, it just does a DD.

I'm sure most of the Linux users here would much rather use dd and (G)parted anyway and do it manually.

Remember I'm just trying to make it easy for the 'basic' user.

HansH
Posts: 214
Joined: Mon Sep 05, 2011 7:49 am

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 5:54 pm

Jaseman said:


Turning the monitor can be a problem for some people.


If that is the case you should not get a Raspberry Pi at this time.....

davidgoodenough
Posts: 74
Joined: Wed Sep 21, 2011 11:55 am

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:00 pm

sylvan said:


davidgoodenough said:


sylvan said:


except for one thing...   the size of the device is encoded in the partition table (typically the first sector) and if you overwrite that, things get confusing later.


Actually you can use any of the parted family (parted itself, or gparted or qtparted) to fix the partition table after you make the copy.  This only works if the device is bigger than the image, but it does work - I do it all the time.


Is that before or after the kernel reads the incorrect size from the partition table?


You would do it on the host machine where you are creating the SD card before you boot.  If you boot with the wrong size, the kernel will see the partitions that are defined, i.e. the small ones.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:25 pm

davidgoodenough said:


sylvan said:


Is that before or after the kernel reads the incorrect size from the partition table?


You would do it on the host machine where you are creating the SD card before you boot.  If you boot with the wrong size, the kernel will see the partitions that are defined, i.e. the small ones.


That would be "before."    And would explain the behavior I've seen.

It may be an issue if the SD card is written on windows (ala the purpose of this GUI tool).  The tool may need to do a fixup in the MBR after writing the image and before exiting.  It may also need a "wipe MBR" advanced function in case the fixup did not happen for some reason.

What is a fixup?

Restoring the original geometry values after writing the image to the card.  If the card is large (and right now I do not remember the lower boundary) then the values are "special" and mean "ignore me and look at the LBA instead."  If the original card is small it gets tricky, see the fdisk source code in the util-linux package to see how all that stuff gets calculated.

Why a fixup?

If the size encoded in the partition table is wrong, the geometry implied (or explicit for small disks) in the partition table will be wrong, and partitioning tools and the kernel can become confused.  It will probably work fine, until or unless you try to repartition the card either to add a partition or change the size of a partition.  At which point it can get messy.  I've seen problems as simple as the partitioning tool refusing to touch the disk to as bad as overlapping partitions being created (which wasn't at all obvious or even apparent until later when data started getting corrupted).

Edit:  BTW, all this is why originally I was saying the tool should partition, and then write filesystem images into the partition(s) instead of using an image of the entire device.  Two partitions (FAT and ext{something}) and either two separate images or else the files for the FAT partition and the image of the ext partition.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:35 pm

I've got an 8gb and 16gb card I can test it on. I wont be releasing this till i've tested it on the Pi as I don't want to get bad feedback surrounding the tool.

The code will also be up for review. My father suggested I use some kind of style checking tool too so I'll hopefully have nice tidy code that is 'to standard'… something I want to achieve as a mostly self taught developer.

Once the Initial functionality is in I'll be glad to have feedback and feature suggestions

EDIT: In reply to your edit. I've directly spoke to Eben and Liz about the SD card images and Eben has said "we generate SD card images by creating an empty file and mounting it via loopback as a block device; these are then written to the physical card using dd ". Liz said "Ideally, yes, you'd need to resize partitions, but Eben says that in practice people can probably do it themselves. And yes, the images just need to be DDed onto the card."

My plan is to take the official image & just make a load of others that can be dd'd by resizing the partitions on that one (taking into account that not every 4gb is 4gb, might be 3.9gb and so on...)

User avatar
jojopi
Posts: 3354
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:37 pm

sylvan said:


Liam Fraser said:


The size of the actual device is not stored in the partition table...


You are wrong about that.

The table stores the size of the device and the size of each partition.


The device size is not stored in the partition table.  See the layout at http://en.wikipedia.org/wiki/M.....oot_record

Even if the device size were stored in the MBR, Linux would ignore it.  It already has to get the device size from the hardware in order to support the case where you put a file system on the whole disk with no partition table ("superfloppy" format).

Even if the device size were stored in the MBR and Linux did not ignore it, it would still be perfectly safe to dd a 4GB image onto an 8GB card, say, and optionally resize it later.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:41 pm

jojopi said:


The device size is not stored in the partition table.  See the layout at http://en.wikipedia.org/wiki/M.....oot_record

Even if the device size were stored in the MBR, Linux would ignore it.  It


Correct, in an MBR partition table it is not explicit.  It is explicit in a GPT.

However the size (and geometry) are implicit in the partition table entries and incorrect values are NOT ignored, they are trusted because they are the data on the disk.  The only time they are ignored is if they specify addresses beyond the physical capacity of the media, and then only the excess is ignored.

You can see for yourself either by looking at the source code for the kernel and util-linux mentioned earlier, or by lots and lots of trials.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:43 pm

jojopi said:

It already has to get the device size from the hardware in order to support the case where you put a file system on the whole disk with no partition table ("superfloppy" format).
Well that's not really correct either.  Look at a volume boot record (the first sector of a floppy or of a partition) and tell me what metadata you find there.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Wed Jan 25, 2012 6:45 pm

jojopi said:


Even if the device size were stored in the MBR and Linux did not ignore it, it would still be perfectly safe to dd a 4GB image onto an 8GB card, say, and optionally resize it later.



Maybe.  Or maybe not.  As I posted (probably while you were posting this) it will most likely operate just fine until you try to resize it later.  And even then it might be fine.  But if the kernel and/or the partitioning tool get confused between data from the device and data from the incorrect partition table, you might not know it until it is too late.

User avatar
jojopi
Posts: 3354
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 4:31 am

sylvan said:

But if the kernel and/or the partitioning tool get confused between data from the device and data from the incorrect partition table, you might not know it until it is too late.
The MBR does not specify the whole drive size.  (And Linux always uses LBA in preference to CHS, although there is no reason the HS geometry would vary with card size anyway.)

So the partition table copied from a fully allocated 4GB card is literally identical to the partition table made on a 8GB card with the same partitions covering only the first 4GB.  (Modulo the disk identifier.  Try it.)

There is nothing "incorrect" about merely having unallocated space at the end of a device.  And nothing confusing about using that space later.

kme
Posts: 448
Joined: Sun Sep 04, 2011 9:37 am

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 4:49 am

jojopi said:

The MBR does not specify the whole drive size.  (And Linux always uses LBA in preference to CHS, although there is no reason the HS geometry would vary with card size anyway.)
So the partition table copied from a fully allocated 4GB card is literally identical to the partition table made on a 8GB card with the same partitions covering only the first 4GB.  (Modulo the disk identifier.  Try it.)

There is nothing "incorrect" about merely having unallocated space at the end of a device.  And nothing confusing about using that space later.


This is correct. There is no problem.

User avatar
Chromatix
Posts: 430
Joined: Mon Jan 02, 2012 7:00 pm
Location: Helsinki

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 5:06 am

Even better idea: make the standard image 3GiB, and use whatever space remains as a home partition.  That avoids the mess of having to resize the system partition just to use the rest of an 8GB+ card.
The key to knowledge is not to rely on people to teach you it.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 7:43 am

That is a good point. However, I think that having a 3gb root partition, and having home as a seperate partition means there could be quite a lot of unused space on the root partition. Remembering that this is targeting fairly basic users (or those in a rush), I think it would be best to have home on the same partition and give everyone as much space as possible.

Anyone wanting to change this will know about partitions anyway so it would be fine I think

Cheers,

Liam.

Neon22
Posts: 29
Joined: Mon Sep 19, 2011 4:11 am

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 9:17 am

Sounds like you've thought it out really well and it looks like its going to work perfectly.

Cheers...

bredman
Posts: 1415
Joined: Tue Jan 17, 2012 2:38 pm

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 9:53 am

I am a great believer in having a separate home partition. It is essential in disaster recovery.

If you think of the typical usage scenario, a teacher may need to be the admin for a dozen students. You should assume that at least one student will go messing around in a dangerous directory such as etc and get stuck. There must be an easy way for the teacher to reimage the system files without destroying the student's files.

I agree however, that a separate home partition causes headaches. Therefore, we need a reliable method for the teacher (or student) to backup and restore the home directory. This must be possible even if the system files on the SD card have been trashed.

Note that the backup and restore should use file operations, not dd, so that the files can be restored to an SD card of a different size than the original card.

If we have a backup/restore tool for the home directory, then there is no need for a separate partition for home.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 1:17 pm

Hey Guys Just popped on to see all this talk about partitions.

Think of a partition as a ice cream tub and the space within it is able to store items, only you have square blocks (filesystem) each block is a box that contains items (data). Box inside a Box inside a Box.

The first 512K bytes of the device hdd is known as sector 0 which contains the mbr partition information. Wiping this would wipe all reference to partitions which contain the filesystem.

The mbr (master boot record) is to help the bios select a boot device that contains the filesystem to boot os/kernel from.

A filesystem sits in the partition (not exactly the same size).

I have linux on a usb boot device atm here are the partitions

typefilesystemMountpoint Size on Disk

Pext2/3.3gb

E

S swap/swap512mb

Pext2/home2.2gb

This is the best way for system recovery as 3.3GB is needed to backup system using DD.

DD can be used to restore a filesystem to a partition or restore a partition to a drive wiping all other partition information in sector 0.

It is possible to restore a 2gb dd with partition information image to a 8gb device, it will create the partitions and filesystem.  For instance if the 2 gb image has the following partitions

Pext2/1gb

E

Sswap/swap512mb

Then once dd'd to the 8 gb would show on it

Pext2/1gb

E

Sswap/swap512mb

Both devices would have un-partitioned space remaining. which can be partitioned and filesystem created and mounted as /home.  As sector 0 also shows the end point of each partition as a number or position on device.

I would guess the pi has

Pfat LBA512mbNo. 1

ENo. 2

Pext2/1gbNo. 3

Sswap/swap512mbNo. 4

As you should know you can have only 4 primary partitions on a device

And 4 primary partitions in an extended volume.

Max primary partitions theoretically are 16.

I am also guessing that the pi rom would come with no user information.

So the home folder would be built on user creation.

I like the idea of /home being a partitioned filesystem

Only the released rom is for first install and not redundancy, so it would not partition the home folder and it would most likely wipe the partition reference for /home partition in sector 0 when re-written.

A subsequent dd image can be created to backup the filesystem in partition 3.

Also an dd image can be created to backup the home directory (user partition).

It would be better when setting the partitions on a sdcard formatted to fat32 lba 512k bs, to wipe the data cleanly using dd first using the standard

dd if=/dev/zero of=/dev/mmcblk1 bs=512

then

dd if=/image.dd of=/dev/mmcblk1

then used parted to create the partition and filesystem ext2 mounted at /home.

This way doesn't matter what size of sdcard you use as long as it has 2gb or more (1.8 1.9 etc).

If you need more space for installed programs, you can always create a partition with ext2 filesystem and mount it at /etc.  and copy the contents of /etc to the the partition then remove the folder /etc from the system filesystem.

This is why i like linux.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Thu Jan 26, 2012 5:25 pm

Thanks for your input marc.

In terms of backup and PiCard, I'm not seeing it as an important feature yet. The actual flashing process will take priority. I was planning to just do a full disk backup using DD. E.g if the user has it the way they want it and wants to back it up before trying a new distro or something like that. The reason is that I'm limited by what I can do on Windows and Mac.

I personally think that any type of home directory backup should be done from the Pi, or from another Linux box (Teachers Pi with a card reader perhaps). Why not just tar the files up or compress it with some other tool. The idea of backing up home is to save the files there rather than save how your operating system was. (I know that .bashrc and similar are all in there though).

User avatar
esbeeb
Posts: 152
Joined: Sun Feb 05, 2012 12:23 am

Re: PiCard - GUI SD Preparation Tool

Sun Feb 05, 2012 12:43 am

Liam,

I appreciate how you are trying to find a solution which is cross-platform: Windows, Mac and Linux.  But you're stuck because you don't have a decent ext4 utility for Windows and Mac.  Fair enough.  Here's a possible trick to get around that.

Oracle VirtualBox IS cross-platform.  Why not make a VirtualBox Virtual Machine with some sort of Linux Distro as the guest OS (perhaps some really minimal, small one).  Your Picard utility would be the only thing that it performs, when it's booted (no time wasted with creation of users, installing software, etc. in that VM's OS).  The ext4-related utilities Picard needs are easily available in the Linux Guest OS.  Hopefully, that VM can take over control of the SD Card to the extent necessary for Picard to "do its magic".

This way, you're only developing your utility for one platform (the VM's guest OS), and not three (Windows, Mac, and Linux).  Then you post the Picard VM for download.

Sure, people would have to do a few extra steps to download and install VirtualBox, but it's dead easy for them, and would make your life alot easier, since you wouldn't be stuck in your current "dead end".  In fact the Raspberry Pi-realted Video tutorials for installing Virtualbox already exists.

This idea is simply just using VirtualBox for yet one more purpose: Picard running inside a Linux Guest VM.

Cheers,
Esbeeb

Return to “Other projects”