marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 7:46 pm

It doesn"t matter what distro really for the kernel as it has to be able to boot any computer.

Gnome is too big to install on lightweight cd. It will have a basic gui running jre (version compatible with your jdk).

Will include basic gui interface partition manager. Or i could just trim down lxde removing what is not needed then write a script to make the iso.

Unless there is already one available to do this task. As long as it has the option to omit user details and dep for iso creation and does not include installer.

As just want it to be for single use dedicated to flashing for pi sdcards.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 8:21 pm

I suggest something like fluxbox as the GUI, I'll leave the rest up to you as it's not my forte

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 8:30 pm

All suggestions welcome its a community afterall.

That said. Took a quick look at fluxbox quite impressive. Not going to need the menu system or most of the apps. Would need desktop and links to start anything else that could be usefull.

All ideas welcome.

roelfrenkema
Posts: 105
Joined: Sat Jan 07, 2012 5:17 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 8:50 pm

@marc Funny you would say that, I just installed a debian arm system on qemu with flux as desktop. But you might have a look at XFCE also as it is also small lightweight and very versatile. Was very happy to see debian has PHP5 on its arm repository.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 8:56 pm

xfce bah lol. you need to install compile software and a whole lotta back end stuff for it. its what lubuntu is using for their netbook edition also openbox. perhaps could be usefull?

Too much to choose from. Definately use x86 debian.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 9:31 pm

Not going to start it until i have mapped out all what is needed (plan).

Also need netbook from friends place.

mobeyduck
Posts: 173
Joined: Tue Nov 29, 2011 6:39 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 9:50 pm

marc said:

Too much to choose from. Definately use x86 debian.

I hope your not talking about the R-Pi there since its ARM and not x86

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Sun Jan 08, 2012 9:59 pm

X86 for pc would work on 64bit machines lowest spec would be 486 64mb ram.

This is for the software to flash pi arm linux sd card in an liveiso linux.

would also be able to still flash pi-cards after win 7 meets its natural and neutral end the people who use linux such as ubuntu wont really need liveiso linux.

it can just solve a few issues that could arise in windows.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 3:01 pm

marc said:


X86 for pc would work on 64bit machines lowest spec would be 486 64mb ram.

This is for the software to flash pi arm linux sd card in an liveiso linux.

would also be able to still flash pi-cards after win 7 meets its natural and neutral end the people who use linux such as ubuntu wont really need liveiso linux.

it can just solve a few issues that could arise in windows.


Hi Marc. Been thinking through the LiveCD solution and there are couple of issues I can see with it.


My program will be downloading the SD images... so where will they be stored?
Even if they are stored I then need to extract them
This means we need NIC drivers, DHCP and all that stuff

The current solution I plan will definitely work. The only problem with it is that it will require a good amount of temporary space because there will need to be an image roughly the size of the SD card to DD. That means a 32gb SD card needs at least 32gb of free hard disk space. Obviously this will need to be more because of OS swap and stuff like that. The only possible way around that is to do the DD in parts - using the seek and count options. Maybe just do it in 1GB or 500MB increments (would still need space for the compressed image, the extracted image and then space for the increments)? Again though... this just makes it more complicated.

Would like opinions on the disk space issue?

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 3:56 pm

Think the .img file needs to be made smaller without the 'free space' and a script at first boot to setup the required swap space partition at the end of the card with a least a 1MB buffer between partitions the start of first partition and after last partition.

dd will copy freespace as well as none freespace it blockwrites so if there is unregistered data on the block it gets copied as well even if the data is not active or accessible.

Problem with linux is that you cannot install to the same media that your installing from because of the way it loads from a kernel and mounts the device.

I would think there is a way to dd the data and arg out the free space and have it set the end point in the partition table seeing as partition reference is only that, a reference of start and end point name value id type and such like written to sector 0.

I have already looked into modifying the img files and shrinking them to a more comfortable and generic size that could be placed onto the cd with the program.

Also i have found an .img flash software for sd cards that is open source and free from pendrivelinux.

Although if you place an sdcard into a usb adapter well.. it gets treated as an usb-hdd, by the system.

Software that is normally used to create backups of partitions could be used to shrink the file size of the img to flash.

dd is a good tool to flash the sdcard.. but to make the img file prob not the best unless someone knows how to just take the data and leave the free space.

there is a small program that can be used to expand file system to partition size and i believe its included in debian core.

1.  Create the partition on SD card keeping to the buffer zone and having an swap file of 512MB partition.

2. dd img to partition (the img will be data/file system only arg partition information).

2.b.  Mount sdcard

2.c.  chroot into sdcard and expand the partition.

3. Expand file system to partition.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 4:09 pm

As for the networking issue.

Live CD connects to the internet even on the wireless, for those like me who don't use ethernet its useful.

The space issue again.

Its possible to take the image file from virtualbox or other vmware type software and mount it in livecd.

if the virtual hdd file is on incremental file size (ie doesn't expand to full set capacity) whatever size that is install with rpi os would be the generic filesize.

Technically could make own distro for rpi to the same specifications as official release.  rebuild the os in our own img so to speak.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 4:53 pm

I would think the images from Raspberry Pi will only be as big as they need and I wouldn't want to tamper with it. I forgot to mention that as long as parted is installed, the Linux version won't have that space issue. The only reason I have to build image files and then DD is because I can't resize linux partitions natively from windows or mac. Linux will use the same parted commands that my 'SD image factory' back end will use. Or they will have the option just to resize it themselves and I'll just download and flash an image.

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 5:07 pm

If the data on an rpi sd card is small enough to fit on a cd with liveboot then there is no issue in flashing the card.  Easier in linux than in windows.

For win users perhaps qemu standalone with basic linux (runs as an app on win32).

I been looking into qemu using qemu manager and there is a way to set a physical hdd to be seen via virtual environment.

Under linux the sdcard is not detected as sdaX its something like mmcblock0.

Your program will still run under this environment and detect the sdcard to flash it.

Maybe you could include the rpi os img in the zip/installer.  One download one click option.  Just a thought.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 5:15 pm

I don't trust virtual environments! What if it doesn't work with a certian drive. That's unfair on them and it's impossible to test. On my Linux box, My sd card shows up as sdx. Either way - I use Fdisk -l so they'll all get listed. I have faith in this solution because I've exprienced all of the utilities I'll be using work well... I just have to glue them all together !

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 5:45 pm

All a bit hackish.. hehe Loving it. Been a while since had to find solutions for people less able to do it themselves.

Windows just isn"t friendly. and it doesn"t play nice with other children.

Be interesting to see what the finish looks like.
The space issue is an issue because anything over 1gb will effect people using data allowance.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 5:53 pm

It won't be a problem for bandwith, just hard drive. The download size will still be almost identical to the official images. I'll be building the other images using the original image and diff files... with 7zip/LZMA compression which is the best I've found. Just hoping that I can build and DD in parts rather than needing 8gb of space for an 8gb image if that makes sense . Yeah it will be very interesting. Once the 20th of Jan comes I can really start hammering it (will have finished exams)

marc
Posts: 48
Joined: Fri Jan 06, 2012 6:40 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 5:59 pm

just out atm buying memcards etc.

sorry. missed what you are doing. how are you creating the required size img to flash memcard?

Ok the zipped file contains the img file and zip normally removes the 0000's out of the file basic principle of compression.  (before getting into overlay compression and same value locational compression.

There must be a way to remove the 000's from the uncompressed img and expand it under windows.  Either case this link if for an image writer for win32:

https://launchpad.net/win32-image-writer/

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 9:23 pm

I'm a bit late to this discussion, I'm afraid, and I must admit I haven't had the time to read all nine pages of posts, but since I lead a small team that has been developing an Ubuntu customization that runs on netbooks from an SD card (Balsa), I thought I should share some of our experiences.

First, it's very, VERY important that users don't accidentally overwrite the wrong device. IMHO a command-line tool is threfore simply too dangerous. We have been using the Ubuntu Win32DiskImager with a fair amount of success. It had a few bugs in it but I and others have been fixing these and a new release was made about two weeks ago. It's Windows-only, but there is an equivalent utility for Linux that has a similar UI (usb-imagewriter). It's written in Python, uses GTK for the UI and dd for the actual writing, so it should be portable to Mac as well (running under X11).

We produce ready-made images for 4 & 8 GB cards, and compress these with 7zip. They have the same files on them, and compress down to the same size. We encourage Linux users to download them with zsync, which allows restarting and provides checksumming, which saves a lot of problems. Unfortunately, I haven't been able to build a Windows version of zsync yet (although I'm sure it's possible) so we tell Windows users just to use their browser. However, we strongly encourage them to run a checksum on the downloaded compressed file, because we don't want to waste time on support requests that turn out to be the result of a corrupted download.

So the set-up sequence for a Windows user is:


Download the image file
Download md5sum, 7zip and win32diskimager
Use md5sum to check the file
Use 7zip to unzip the file
Use the imager to write the file to the card

The sequence for a Linux user is similar, except that the three utilities at step 2 are packages to be installed, and the image writer has a different name (usb-imagewriter).

Like R?, Linux is in a single, large ext3 partition, and this requires us to have separate images for each card size. However, we're planning to change this, in one of two possible ways. The simplest solution is to put /home in a separate partition that gets almost all of the free space. It's then possible to resize it from the running system by unmounting /home temporarily, or even doing it automatically on startup if there is unused space on the card. However, with a split system like this there is always the potential for running out of space on the root partition. So another possibility is to have a single partition that gets expanded on startup while Linux is still running out of the initrd (initial ram disk) before it has mounted the root partition. This requires putting parted and resize2fs into the initrd, which would cause considerable bloat, so we would create a single-use initrd that does this and this alone, and switches in the regular initrd just before it finishes, and then reboots.

I think we should avoid reinventing the wheel here. Good card imaging programs already exist, so there shouldn't be a need to write new ones.

It would also be good to collaborate as much as possible. I'm very interested in R?, and since there is considerable overlap with my day job working on our Balsa project I would be glad to contribute something for resize-to-fit on startup. I've already produced a customized, one-off initrd for other purposes in Balsa, so I have a head-start on something for resizing. Also, I use Debian as my main development platform so I should be comfortable with tweaking the R? Debian system.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 9:40 pm

Hi Neil. It would be very interesting to see the Raspberry Pi images resize themselves on first boot. Would be great to see a proof of concept once the official images are out as it would remove a lot of the complexity from my current solution which goes a little bit like this:


Back end that generates card images all the way from the minimum required card size up to 32gb (max supported SDHC)
Use a binary diff utility to create patch files for these images using the official image as the base image (won't be large at all because not much has changed)

My tool will:


Let the user select a distro
Download a 7zip file for that distro with the original image, with patch files for all different sizes
Verify the download with a hash
Extract the 7zip file
Build the required image and DD it (hopefully I'll be able to do that in stages... I.E make the first 100mb of the output image and then DD it and then the next 100mb and so on... so the user doesn't require a rediculous amount of disk space)
This will work across Windows, Linux/Unix and Mac.
Linux users will be able to resize using the same parted commands that my image generation back end will use (not really a priority)


As secondary features, I'll also be providing:


Image backup and restore
Basic DD flashing of any image


I'm still going to work on this system for now as if nothing else, it is a fantastic experience of using Java which I'll be using a lot of in uni. If you were able to provide images that could resize on first boot that would be the preferred solution and I'd happily and gratefully switch once it was tested

Cheers,

Liam.

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 10:34 pm

I'll do the work for Balsa and let you know how I get on. I'll try to do that sooner rather than later. The main issue is getting enough stuff into the initrd (libraries, etc.)

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 10:36 pm

Sounds good Neil!

Good luck with your project. I'm pretty confident in mine... so no rush on the images

Cheers,

Liam.

HB
Posts: 43
Joined: Sun Jan 01, 2012 9:49 am

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 10:59 pm

Liam Fraser said:


Hi Neil. It would be very interesting to see the Raspberry Pi images resize themselves on first boot. Would be great to see a proof of concept once the official images are out


If you want to see a proof of concept now, the Ubuntu images for the PandaBoard do this.

My tool will:

Build the required image and DD it (hopefully I'll be able to do that in stages... I.E make the first 100mb of the output image and then DD it and then the next 100mb and so on... so the user doesn't require a rediculous amount of disk space)


Unzipping and writing on the fly can be done easily enough with basic Unix utilities... "gunzip -c image_name.img | dd of=/dev/whatever" (example using gzip as I'm not familiar with 7zip's command line options off the top of my head). Doing this in Windows might be a bit tricky.

User avatar
jojopi
Posts: 3353
Joined: Tue Oct 11, 2011 8:38 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 11:20 pm

Neil Mayhew said:

So another possibility is to have a single partition that gets expanded on startup while Linux is still running out of the initrd (initial ram disk) before it has mounted the root partition. This requires putting parted and resize2fs into the initrd, which would cause considerable bloat, so we would create a single-use initrd that does this and this alone, and switches in the regular initrd just before it finishes, and then reboots.

I do not believe an initrd is necessary.

On the first boot the script can rewrite the partition table so that the root partition has the same start sector but a greater length.  The kernel will refuse to acknowledge the change because the filesystem is busy.

On the second boot the partition will be bigger and the resize2fs can be issued online.

Optionally you can force the reboot between the two stages.

Apparently parted dropped support for resizing and moving partitions after version 2.4.  (And these functions were never safe on mounted filesystems anyway.)  So it is probably necessary to remove the partition and create the new one with the same start sector.  Of the partitioning tools I have looked at, I think that sfdisk has the safest interface for this.

Alternatively, if we are happy to restrict to four partitions and a fixed set of card sizes, it would be possible to install a precomputed MBR using dd.

Neil Mayhew
Posts: 9
Joined: Thu Nov 03, 2011 6:15 pm
Contact: Website

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 11:37 pm

Most of what I've read elsewhere says this won't work. If the kernel is using the file system, even read-only, it's not safe to change the partition table, and certainly not safe to resize the file system. For example, the resizing utility itself could get moved as it is running, and when the kernel needs to page some part of it into RAM, garbage will get loaded instead of the original code.

This is why a reboot is forced if repairs are done to the root file system during startup. Although fsck isn't run from the initrd, the root fs is still read-only at that point, and a repair is typically way less drastic than a resize. For example, it's unlikely that fsck or it's dependent .so's will get moved around or changed.

I definitely don't recommend resizing a mounted filesystem, or changing the partition table of a disk on which filesystems are currently mounted.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: PiCard - GUI SD Preparation Tool

Mon Jan 09, 2012 11:55 pm

Neil Mayhew said:


Most of what I've read elsewhere says this won't work. If the kernel is using the file system, even read-only, it's not safe to change the partition table, and certainly not safe to resize the file system.


Nonsense.

It is not safe to relocate a partition.  But modifying the partition table to fill the available space is safe enough.

And check out the resize option to the mount command.  It will resize using only the kernel, no need for additional utilities.  It is also safe enough, much safer than using some resize utility.

Return to “Other projects”