PiCard - GUI SD Preparation Tool


269 posts   Page 1 of 11   1, 2, 3, 4, 5 ... 11
by bradburts » Tue Dec 27, 2011 10:24 pm
"…to boldly program where no one has programmed before…"

PiCard is an easy to use SD card preparation utility for the Raspberry Pi.
PiCard may be used to install the official and other Linux distributions onto a Raspberry Pi SD card.
PiCard may be run on a Linux PC or from a Windows or Mac PCs using VirtualBox or a Linux Live bootable USB/CD.
PiCard optimises the SD creation process producing the fastest SD card burns possible.
PiCard is free and open-source software.

This topic has been created as a meeting room for the authors and those who wish to contribute to the project.

Operating scenarios comming soon!
Posts: 341
Joined: Sun Oct 02, 2011 7:07 am
by mengel » Tue Dec 27, 2011 11:42 pm
So, this is going to do the partitioning and write the gpu blob to the card and then the OS is written in another step?
Posts: 52
Joined: Thu Oct 27, 2011 1:51 pm
by bradburts » Tue Dec 27, 2011 11:50 pm
The utility will automate the process of generating a bootable SD image.

Download, partitioning, filesystems, GPU blob and OS image with one click.
Posts: 341
Joined: Sun Oct 02, 2011 7:07 am
by JonHubertBristol » Wed Dec 28, 2011 12:08 am
Will it also, align the blocks on the SD card to the filesystem? (forgive me if I stated that wrongly) I just saw a thread on Optimisig Linux for flash memory...
Posts: 11
Joined: Wed Oct 26, 2011 9:10 pm
by mengel » Wed Dec 28, 2011 12:10 am
Cool ! Kinda like unetbootin does with usb sticks.
Posts: 52
Joined: Thu Oct 27, 2011 1:51 pm
by liamfraser280 » Wed Dec 28, 2011 12:20 am
Here's a little picture of what we have in mind :)

Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
by OrbDemon » Wed Dec 28, 2011 10:43 am
Nice idea :)
Posts: 11
Joined: Sat Dec 24, 2011 9:00 am
by bradburts » Wed Dec 28, 2011 1:51 pm
Thanks for the words of encouragement all.


Will it also, align the blocks on the SD card to the filesystem? (forgive me if I stated that wrongly) I just saw a thread on Optimisig Linux for flash memory…


Post the link please!
Posts: 341
Joined: Sun Oct 02, 2011 7:07 am
by Reiep » Wed Dec 28, 2011 4:15 pm
That's a great idea, i would participate with pleasure :)
Posts: 2
Joined: Mon Oct 03, 2011 2:54 pm
by abishur » Wed Dec 28, 2011 4:16 pm
Lookin' good!  I know this is way outside your goal but it would be cool if it was java, then it could run on any OS.

Fortunately though, I use Linux! :P :P
Dear forum: Play nice ;-)
User avatar
Forum Moderator
Forum Moderator
Posts: 4261
Joined: Thu Jul 28, 2011 4:10 am
Location: USA
by bradburts » Wed Dec 28, 2011 4:35 pm
Thanks for the support.

We are using Python so it should all be portable.

Please note the process will work with any PC. PiCard will initially be stuck on Linux. The reason for this is that we have not found a PC side Ext3/4 library yet.

If anyone can point us to a Windows Ext3/4 library then that would be great!

In the first instance Linux will be provided by VirtualBox.

We will then look to launch PiCard from a LiveCD.
Posts: 341
Joined: Sun Oct 02, 2011 7:07 am
by jojopi » Wed Dec 28, 2011 5:37 pm
You should consider whether it would be less work, and make it easier to support Windows, if you pre-prepared compressed image files for each common size of flash card.  Then no actual partitioning and formatting would be required at "burn" time.

Better still, the distros should release images for the smallest card size that they support, with scripts included that can grow the important filesystems after booting.  This is not difficult to do, and would not absolutely require LVM.

Regarding alignment, the modern consensus appears to be to align every partition start to an LBA multiple of 2048 (1MB).  I doubt that it is worth worrying about the actual write and erase block sizes.
User avatar
Posts: 2036
Joined: Tue Oct 11, 2011 8:38 pm
by JonHubertBristol » Wed Dec 28, 2011 6:19 pm
Bradburts said:


Thanks for the support.

We are using Python so it should all be portable.

Please note the process will work with any PC. PiCard will initially be stuck on Linux. The reason for this is that we have not found a PC side Ext3/4 library yet.

If anyone can point us to a Windows Ext3/4 library then that would be great!

In the first instance Linux will be provided by VirtualBox.

We will then look to launch PiCard from a LiveCD.


Maybe this link might help in making it available for windows without the virtual machine http://www.andlinux.org/index.php
Posts: 11
Joined: Wed Oct 26, 2011 9:10 pm
by JonHubertBristol » Wed Dec 28, 2011 6:23 pm
Bradburts said:


Thanks for the words of encouragement all.


Will it also, align the blocks on the SD card to the filesystem? (forgive me if I stated that wrongly) I just saw a thread on Optimisig Linux for flash memory…


Post the link please!


Here you go :)  http://www.raspberrypi.org/for.....8;search=1
Posts: 11
Joined: Wed Oct 26, 2011 9:10 pm
by liamfraser280 » Wed Dec 28, 2011 10:06 pm
jojopi said:


You should consider whether it would be less work, and make it easier to support Windows, if you pre-prepared compressed image files for each common size of flash card.  Then no actual partitioning and formatting would be required at "burn" time.

Better still, the distros should release images for the smallest card size that they support, with scripts included that can grow the important

filesystems after booting.  This is not difficult to do, and would not absolutely require LVM



Abishur said:


Lookin' good!  I know this is way outside your goal but it would be cool if it was java, then it could run on any OS.


You have both made great points which were considered at the start of development. Keep in mind that this will be an alpha release initially. A couple of reasons why we are going with Python:


  • Myself and Pete both knew Python and where we didn't, it's very easy to learn.

  • The Foundation seem to like Python, so a little bonus there

  • Python teamed with VirtualBox is allowing us to quickly develop a rough solution (that will be safe to users because it is contained in VBox) that with proper documentation, will be very easy (although maybe slightly time consuming in terms of downloads) to use.


More important is why we are sticking Linux only for now:


  • We don't know anything about the SD preperation process (as of now, currently waiting for the detailed, exact instructions from Eben and Liz :) )


  • Note: We will be using these instructions exactly so we are following official protocol (if block alignment isn't on there then we won't do it for now)


  • Speeding up testing

  • We do kind of support win/mac through VirtualBox so not too bad


  • There are a few ways SD card preperation could happen:

    Method 1


      1. Create partitions

      2. Copy or DD gpu firmware to the first partition

      3. Get root FS as a zip file (like on beagleboard) and extract that to an ext3/4 partition (very tricky from Windows and Mac)


      Advantages:



      • Partitions are already the correct size



      Disadvantages:



      • Need to create an ext3/4 partition from windows and mac

      • Need to then proceed to mount the ext3/4 windows and mac and copy files to it

      • Both of these should be done reliabley without commercial tools... very difficult



      Method 2


        1. Create partitions

        2. Copy or DD gpu firmware to the first partition

        3. DD root FS to second partition

        4. Use a partitioner back end such as parted to resize the second partition to take up the unallocated space on the SD card


        Advantages:



        • Hardly any. On similar par to the first method because we still have to resize the free space



        Disadvantages:



        • Need to create an ext3/4 partition from windows and mac

        • Need to resize partitions when we are finished

        • Both of these should be done reliabley without commercial tools... very difficult



        Method 3 (Ideal Method and jojopi's suggestion)


          1. Either have:


          1. Both partitions & partition table in one image

          2. On first boot the partition will be resized to take up rest of the free space

          3. (Any custom changes can be done by the user with Gparted for example... but that is outside of our tool)


        1. Or:


          1. Somehow create a partition table

          2. DD First Partition (GPU stuff and so on)

          3. DD Second Partition

          4. On first boot the partition will be resized to take up rest of the free space


          Advantages:



          • There is DD for windows, mac and linux

          • No need to access or resize any linux file systems, the Pi does it all for us :D ... happy days :D

          • Do able!



          Disadvantages:



          • GPU firmware and rootfs images might want to be kept seperate

          • If so would require some way to create a partition table that would be free, preferabley open source and work across all OSes (and I'm no professional programmer... Yet ;) )




          I think the multiple disk images for each distro would be very tricky. We want to try and support as many distros as possible. Say we have 2, 4, 8, and 16 gig card sizes as standard. Multiply that by how many distros, and then also we have to consider updating the images every so often.

          With regards to Java... and therefore not needing to download a huge (roughly 200mb in a zip archive) Virtual Machine & VirtualBox to do the task... which is a lot but it's also safe and if they can download a distro - it's not too much more...

          If we find out that Method 3 is being used, especially the version 1 where we literally have to download an image and DD it. I'd be very happy to do a full rewrite in Java, once I have some spare time after January college exams. How is the java UI looking across different platforms? An Ugly GUI is still a GUI though :P . If I was a few years wiser/older I'd do it in C++ to reduce the dependancies, then again would either have to write native 3 front ends or use a Graphics toolkit. I have done little bits of C and C++ but I don't know enough to be able to produce reliable, solid, proper code in it... Yet ;)

          Hope I have explained our choices well.

          Cheers,

          Liam.

          Posts: 354
          Joined: Tue Oct 04, 2011 6:53 pm
          by liz » Wed Dec 28, 2011 10:11 pm
          Eben's meant to be emailing you about this with the info you asked for (it is crazy busy around here at the minute) - he has three emails he is not allowed to go to bed without sending tonight, and yours is one of them. ;)
          --
          Head of Comms, Raspberry Pi Foundation
          User avatar
          Raspberry Pi Foundation Employee & Forum Moderator
          Raspberry Pi Foundation Employee & Forum Moderator
          Posts: 4082
          Joined: Thu Jul 28, 2011 7:22 pm
          by liamfraser280 » Wed Dec 28, 2011 10:18 pm
          liz said:


          Eben's meant to be emailing you about this with the info you asked for (it is crazy busy around here at the minute) - he has three emails he is not allowed to go to bed without sending tonight, and yours is one of them. ;)



          Great stuff Liz :) ! Ideally we'll end up at a point where the tool will not need VirtualBox but the chances of that becoming possible will be a lot clearer once I have the email from Eben. I'm exhausted after thinking through & typing my previous post haha! Off to relax for a but but I'll have my phone - egarly awaiting Eben's response.

          If you wouldn't mind Liz i'd really appreciate you showing Eben my previous post. So he can see what my thoughts on the matter at this time are, and he can clear some of them up?

          Thanks,

          Liam.
          Posts: 354
          Joined: Tue Oct 04, 2011 6:53 pm
          by mkopack » Wed Dec 28, 2011 11:02 pm
          Liam,

          GUI's on Java have gotten a LOT better since the original AWT stuff from JDK 1.0.2... (AWT tried to use the native system's GUI widgets, the newer Swing library actually draws components based on Look and Feel libraries.) The Swing toolkit has allowed for a lot better cross-platform GUI's, and if you use it properly, along with the proper Look&Feel option for your particular OS, you can get stuff that looks very native but still portable (just swap the L&F libraries, but you often have to deal with different widget sizes and fonts and such, so you have to be careful in laying components out properly.)

          A lot of the time, though, (and this is what I'd suggest), people use the Metal L&F. That way every platform would have an identical looking UI. If you keep the UI fairly simple (a couple buttons, maybe a text entry field or two, etc.) it should be pretty straightforward to have a java-based UI app that would work the same on all platforms.

          I'd offer to help, as I'm a professional java developer, but I'm also in grad school on top of my day job, and about to start up a heavy workload quarter (ugh! shoot me now!) But I can certainly offer some suggestions if you want to run anything by me...

          --Mike
          User avatar
          Posts: 242
          Joined: Mon Nov 07, 2011 8:46 pm
          by mobeyduck » Wed Dec 28, 2011 11:06 pm
          have you ever looked at universal usb installer

          Universal USB Installer is a Live Linux USB Creator that allows you to choose from a selection of Linux Distributions to put on your USB Flash Drive. The Universal USB Installer is easy to use. Simply choose a Live Linux Distribution, the ISO file, your Flash Drive and, Click Install. Other features include; Persistence (if available), and the ability to fat32 format the flash drive (recommended) to ensure a clean install. Upon completion, you should have a ready to run bootable USB Flash Drive with your select Linux version installed.


          They use casper.rw files to make you able to write things to the usb drive, mabey some thing to look at while creating source code is available at the right hand side of the download button
          Posts: 173
          Joined: Tue Nov 29, 2011 6:39 pm
          by bradburts » Wed Dec 28, 2011 11:38 pm
          jojopi said:


          You should consider whether it would be less work, and make it easier to support Windows, if you pre-prepared compressed image files for each common size of flash card.  Then no actual partitioning and formatting would be required at "burn" time.

          Better still, the distros should release images for the smallest card size that they support, with scripts included that can grow the important filesystems after booting.  This is not difficult to do, and would not absolutely require LVM.


          @ jojopi

          Which tool / library would you use to write an ISO to USB on Windows?

          I think that I should try that approach again.

          Liam & I have some differences on this one ;)

          IMHO The user is will be unfamilar with Linux. If they can Linux then they probably won't need PiCard. The target audience is therefore Windows and technically naive.  Yes others will see benefit but they will not need PiCard.

          Assuming that this is so then these are the releative benefits for the Windows & naive user.....

          1) VirtualBox - Booting PiCard from VirtualBox

          VirtualBox requires some setup and configuration and is therefore not an optimum solution for the target audience. Intimidating is a word.

          A quick win but I do doubt that we are gaining much.

          2) Live USB/CD imager - Bootstrapping PiCard from a Live USB/CD.

          This option could reduce the amount of end user input and configuration. At its simpliest we would hard wire a Live USB creator and so cut interaction down to a minimum. The user would be told exactly what to do at each step. All choice would be removed.

          If we can program an ISO image of a Live USB hosting PiCard using Windows then this method is as 3) except that in this option the end user needs two cards and the user must reboot before PiCard runs.

          3) Program Rasberry ISO image to USB directly from Windows

          The end user launches a program on Windows and the Raspberry Pi SD card is programmed in one step using a prepared ISO.

          This is the simplest solution albeit (possibly) at the expense of having to create a number of ISO images, one for each considered size of SD card.

          It would be fairly simple to create a back office system to generate the various ISOs.

          The customised ISOs would need to be hosted and whilst I would be happy too I don't think that my hosting options would take the strain.

          Not a huge issue though.

          @mobeyduck

          Thanks. These tools usually create a disk with Linux on FAT. We need a disk with a FAT partition followed by Linux partitions.

          I will run your links down & make sure :)
          Posts: 341
          Joined: Sun Oct 02, 2011 7:07 am
          by liamfraser280 » Thu Dec 29, 2011 12:30 am
          Hey Pete,

          I do agree that windows is the main target platform. Having said that, people who follow my video tutorials didn't seem to have a problem with VirtualBox and some of them have never used Linux before. With the proper documentation it shouldn't be too intimidating.

          I've just got an email from Eben with some SD card stuff, I've replied asking for a little more info and CC'd you so check that out.

          The solution we have in mind at the moment is obviously very bulky but the best logical solution with what we know right now. It would be nice to have a working, albeit a little messy and rough solution for the less techie people who buy a Pi now. Better than a terminal for most of them I'd think.

          I think I have a new solution in mind, it's based on your Solution 3 above and I think we should seriously consider it if we can get it working:


          • No matter what format the official images come in we will make our own Pi Card images for all of the official distros that have the following:


          • Partition Table

          • FAT32 partition with GPU firmware and Linux Kernel

          • ext3 partition and swap partitions (or an alternative using one ext3 partition with ext3 images and a swap image... see the email from Eben)


        2. What will then happen is, we have a custom script that is run at first boot that will resize the ext3 partition and swap partitions to take up the rest of the disk. Not sure how to go about that but I think it's possible

        3. We could also in the future, allow advanced users to customise their partitions by having a config file on the FAT32 partition that our script or utility reads! Even windows 98 can read and write to that ;)

        4. This means that our actual PiCard utility would literally download an image, and flash it, and maybe write a config file to FAT32 for custom partitioning. Easy peasey - and would work natively


        5. That just means we'd have to come up with some kind of utility or standard procedure for creating our PiCard images if others want to make them. and also the script to resize the partitions.

          What does everyone think?

          P.S Still waiting for more specifics on the SD card images... for all I know they could be in the exact format as spec'd above and automatically resize on first boot! That would be great :D

          Cheers,

          Liam.
          Posts: 354
          Joined: Tue Oct 04, 2011 6:53 pm
          by Tomo2k » Thu Dec 29, 2011 1:43 am
          Wow, this is looking really great.

          As somebody mentioned earlier, it will need to run 'natively' (as in, run an installer and it handles all support gubbins) under Windows 32bit and 64bit platforms as those are the platforms of users who don't know much about the computermachine.

          People already using Linux won't need it - though they'll like it of course!

          Please, continue with your great works!
          Posts: 126
          Joined: Mon Dec 19, 2011 10:00 pm
          by juniordevxd » Thu Dec 29, 2011 2:50 am
          Is there some way for people to contribute to this program....sounds like a good one to wet my toes with (I'm not a noob developer just noob around here, been lurking since like august! :) )
          Posts: 2
          Joined: Thu Dec 29, 2011 2:47 am
          by liamfraser280 » Thu Dec 29, 2011 10:38 am
          I've just set up a github for PiCard, there was already a tool called that so I just named it RaspPiCard

          The python code so far is on there for anyone who want's to look! I don't know much about git yet so please bear with me. I've already moved away from the master branch. The plan is to keep this python version away from the master branch and then when we work out a way to do solution 3 as mentioned above... that will become master.

          https://github.com/liamfraser28036/RaspPiCard
          Posts: 354
          Joined: Tue Oct 04, 2011 6:53 pm
          by liamfraser280 » Thu Dec 29, 2011 1:50 pm
          Just out of interest... who out of the Windows people would prefer a native tool that would be basically a front end to DD and then be shown step by step how to resize their partition with a free tool such as http://www.paragon-software.co.....tures.html - Would probably be easier to follow than installing VirtualBox?

          I'd also be able to develop a much more robust solution this way because I am experienced and very comfortable with using C#. Any system since XP will more than likely already have the .Net framework installed.

          Let's be honest Linux users, if you are already using it.... DDing an image and resising with Gparted is a pretty easy task.

          Not thinking of abandoning the VirtualBox solution just yet as that provides the exact same tool for everyone... thats fair and plus it's half way finished!
          Posts: 354
          Joined: Tue Oct 04, 2011 6:53 pm