Chris Tyler
Posts: 70
Joined: Thu Jul 28, 2011 12:16 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 1:36 am

It would be great to have an easy-to-use tool for creating SD cards for the Raspberry Pi. Such a tool should:
- have a GUI
- be cross-platform - run on at least Windows and Linux
- be capable of downloading (one of several) images or using local images
- auto-detect media

These is a tool called liveusb-creator written by Luke Macken that does almost exactly what we need, except that it writes USB flash drives (installing live CD images on to them). I've asked Luke about adapting liveusb-creator to do Pi SD card setup; he doesn't have the cycles to do this (at least for several months) but is willing to answer questions about the existing codebase if someone wants to take this project on. I'd take it on myself but I'm also swamped out for the foreseeable future.

The changes required to make this a Pi SD installer include:
- changing the media-detect code to detect SD cards instead of flash drives
- adding partitioning logic (probably fixed partition size for the FAT partition, and $restOfTheCard for the 2nd partition)
- stripping out the x86 bootloader installation
- stripping out a lot of the setup logic needed for a live image, including the persistent data logic - the Pi filesystem image should just be dumped onto partition 2 of the SD and then resized to fill the entire partition

Any Python devs interested in taking this on?

The liveusb-creator website is at:
https://fedorahosted.org/liveusb-creator/

(Note: the original tool is Fedora-specific but the Pi SD card tool should be able to do any distro, assuming a FAT first partition and an ext[234] second partition).

kme
Posts: 448
Joined: Sun Sep 04, 2011 9:37 am

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 1:57 am

Isn't that overdoing things a bit? Assuming the source is a bootable ISO image a simple one-line dd does the trick. And I'm fairly convinced there is a dd for Windows too from cygwin.
http://fedoraproject.org/wiki/.....e_Live_USB

User avatar
ukscone
Forum Moderator
Forum Moderator
Posts: 4214
Joined: Fri Jul 29, 2011 2:51 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 1:58 am

I have a program I wrote a few years ago to do pretty much exactly that for the Zipit and a couple of other devices except it's written in euphoria and is console (windows and linux). I'm out of cycles myself until after thanks giving but if no one has knocked anything up before then i'll take a crack at it.

Chris Tyler
Posts: 70
Joined: Thu Jul 28, 2011 12:16 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 2:47 am

@kme - Even a straight dd is dangerous in the hands of a beginner -- choose the wrong device and you'll toast your host system. Plus dd doesn't provide a really nice first user experience.

The idea is to provide a super-simple tool that installs easily and can be used reasonably safely by anyone to produce a working Pi SD card.

User avatar
Montekuri
Posts: 449
Joined: Thu Sep 22, 2011 6:26 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 9:19 am

We can use a USB SD Card Reader.
We put it on the usb port and the tool will "think" it is a USB flash device.

na1pir
Posts: 71
Joined: Fri Sep 02, 2011 4:43 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 11:37 am

Forked ImageWriter (with patches for R-pi) from openSUSE might fit this task.

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 12:03 pm

Let's keep it simple and cheap guys:

@montekuri: I don't think forcing everyone to buy an extra piece of kit is a good idea.
@na1pir; does that software have any advantages over the suggested one ? Does it work in Windows ?

doug
Posts: 28
Joined: Wed Nov 16, 2011 3:51 pm
Location: Llanharan, Wales

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 1:27 pm

Quote from Chris Tyler on November 21, 2011, 01:36
It would be great to have an easy-to-use tool for creating SD cards for the Raspberry Pi. Such a tool should:
- have a GUI
- be cross-platform - run on at least Windows and Linux
- be capable of downloading (one of several) images or using local images
- auto-detect media


That sounds like a good idea, however the tool you later mention is Python based, would that not mean the user would need to install Python including whatever specific generation of Python you use? At least from a Windows perspective that seems to detract from the easy to use aspect or add a lot of overhead by requiring another installer. Can you make all that transparent to the user?

I'm not exactly sure what is the best cross platform system for UI development is, would something like Qt be good for this? If the code is fairly generic it may make sense to write a cross platform tool otherwise it might be worth considering something more bespoke for each platform.

Unless the Raspberry Pi is supplied with an image already on an SD card then the very first user interaction with the Raspberry Pi will be writing an image on to the SD card. So for inexperienced users it needs to be easy with plenty of hand holding and as automated as possible. It might be useful if you could integrate an image with the tool so Raspberry Pi distros could offer an automatic installation tool without the need to acquire or locate the images separately.

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 2:01 pm

@doug: try the current version: it is integrated (no dependencies to install), is OK to use (I've got ideas about the UI, though ^^), and Just Works.
I'm sure there's more elegant and cutting-edge ways to do the same thing. I think the issue is more to cobble together quickly something reliable. Starting off from something way proven, that already does 80% of what's needed, and whose dev is OK to help, does not sound like a bad idea ?

doug
Posts: 28
Joined: Wed Nov 16, 2011 3:51 pm
Location: Llanharan, Wales

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 2:08 pm

Quote from obarthelemy on November 21, 2011, 14:01
@doug: try the current version: it is integrated (no dependencies to install), is OK to use (I've got ideas about the UI, though ^^), and Just Works.
I'm sure there's more elegant and cutting-edge ways to do the same thing. I think the issue is more to cobble together quickly something reliable. Starting off from something way proven, that already does 80% of what's needed, and whose dev is OK to help, does not sound like a bad idea ?

No, not a bad idea, I was just thinking through the user issues that might be encountered. My limited experience of Python so far especially on Windows has shown it can present a few issues. I assume the ideal scenario will be some software you can unzip into a directory, double click an icon and run. After all there is no point simplifying the process of creating an image if that isn't simple itself :-)

na1pir
Posts: 71
Joined: Fri Sep 02, 2011 4:43 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 2:24 pm

@obarthelem-y: well as far as I know, imagewriter needs .net or mono so yes it works on windows as well.
I know there is some work done on openSUSE ARM port, so it might be good idea to contact developers and ask them what do they use for burning images to SD cards. And I guess ubuntu and linaro community can help as well.

User avatar
Montekuri
Posts: 449
Joined: Thu Sep 22, 2011 6:26 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 3:01 pm

Quote from obarthelemy on November 21, 2011, 12:03
Let's keep it simple and cheap guys:
@montekuri: I don't think forcing everyone to buy an extra piece of kit is a good idea.
I think that if I download an image, first I do it in my desktop computer. Or in any Lanhouse computer.
I don't think that every desktop computer has a SD card reader/writer by default.
And I need to burn the image in the SD card in some way. The SD card Reader cost U$3.48.

doug
Posts: 28
Joined: Wed Nov 16, 2011 3:51 pm
Location: Llanharan, Wales

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Mon Nov 21, 2011 3:09 pm

Quote from Montekuri on November 21, 2011, 15:01
Quote from obarthelemy on November 21, 2011, 12:03
Let's keep it simple and cheap guys:
@montekuri: I don't think forcing everyone to buy an extra piece of kit is a good idea.
I think that if I download an image, first I do it in my desktop computer. Or in any Lanhouse computer.
I don't think that every desktop computer has a SD card reader/writer by default.
And I need to burn the image in the SD card in some way. The SD card Reader cost U$3.48.

Or for UK readers, I personally picked up my last two USB SD card readers in Poundland. So for average newbie it's not a huge investment of time, effort and money to pick one up from a local store for a £1 if they're computer is not already equipped with an SD card reader. Enterprising users could even use their digital camera, sat nav or other devices if it uses SD and appears as a mass storage device.

Chris Tyler
Posts: 70
Joined: Thu Jul 28, 2011 12:16 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Tue Nov 22, 2011 6:10 pm

Quote from doug on November 21, 2011, 15:09
Or for UK readers, I personally picked up my last two USB SD card readers in Poundland.

Likewise, USB SD card reader/writers started to show up in Dollarama here (Canada) for $1.50 or $2.00 and they seem to work just fine, so people without an SD slot on their PC/laptop won't be too hard done by. We still need a simple card-writing software solution for them though :-)

Asmodeus
Posts: 29
Joined: Mon Aug 01, 2011 7:44 am

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Wed Nov 23, 2011 1:54 pm

Quote from kme on November 21, 2011, 01:57
Isn't that overdoing things a bit? Assuming the source is a bootable ISO image a simple one-line dd does the trick. And I'm fairly convinced there is a dd for Windows too from cygwin. [...]

dd for windows:
http://www.chrysocome.net/dd


Pseudotsuga
Posts: 2
Joined: Sat Nov 26, 2011 7:18 pm

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sat Nov 26, 2011 7:26 pm

Perhaps it would be easier to simply write up a front-end to dd? In essence have a Linux and Windows compatible piece of software (perhaps written in Java? Then Macs are supported as well, and it's an easy install for everyone) with a simple front end that automatically detects a SD card, allows the user to select an iso image and then runs dd on the iso to the specified card. The problem of inexperienced users potentially erasing their hard drive is solved by simply not giving them the option, or by hiding it under an advanced menu option or something of the sort. I'm not much of a programmer, so I can't really have a stab at this myself, and I don't really know how difficult it would be to implement, but it is another option. All the program would really need to do would be to scan for an sd port and find if a card is in the port, and then pass the location of the iso image and the sd card to dd, and display all of this with a simple gui wouldn't it?

doug
Posts: 28
Joined: Wed Nov 16, 2011 3:51 pm
Location: Llanharan, Wales

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sat Nov 26, 2011 8:16 pm

If all that is being done is copying a file as is to a device I don't see the need for dd for such a trivial task. The most important thing is presenting a simple UI that is foolproof and ensure that the first experience with a Raspberry Pi isn't a wiped hard drive.

Chris Tyler
Posts: 70
Joined: Thu Jul 28, 2011 12:16 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sun Nov 27, 2011 4:10 am

We need more than a blind dd to do this well. Ideally, we want to:
(a) Find what we think is an SD card
(b) Confirm with the user that that's the *right* media (multiple removable-media readers on one system are not unknown)
(c) Let the user select the image they wish to install, and/or download the image for them
(d) Partition the card
(e) Copy the first part of the image to the first partition on the card (FAT), and the second part of the image to the second partition of the card (ext2/3/4) (the image file will need to have two parts: tar or zip format perhaps?)
(f) Resize the second partition image to fill the SD card

This accommodates different card sizes, avoids writing excessive data to the card (i.e., the empty part of the filesystem), and presents a nice/safe UI to the user.

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Thu Dec 22, 2011 10:09 pm

Has anyone started anything like this yet? I might not be able to do much untill the end of january but I would be up for giving it a try.

I'd rather start from scratch though rather than adapting this tool.

Also, I personally think that if we get a generalised set of steps that uses external tools, I.E DD and some kind of tool to list storage devices and partitions, and finally a partition resizing tool. Then we could write front ends seperately. I.E one written in .Net for Windows because anyone with an XP install will probably have at least .Net 2.0... saves them having to download Python, or Java. Then again the Linux people should get a nice gui too, so I'll have a think about language and GUI toolkit etc...

Does anyone know of an ext3 or ext4 partition resizing tool for windows that isn't a fully blown partition manager?

The rest seems possible, just the partition resizing that concerns me :/

If not how about writing this all on Linux, and then having a LiveCD with the tool for people using Windows. I don't know much about mac: Can it boot a LiveCd?

liamfraser280
Posts: 354
Joined: Tue Oct 04, 2011 6:53 pm
Contact: Website

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Thu Dec 22, 2011 11:13 pm

Another way to do it could be to DD a default image. Then provide a tool like Gparted that would work something like:
1.Set desired partitions and click apply
2. Kill all unnecessary processes
3. Unmount sd card
4. Resize sd card partitions
5. Remount SD card

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Thu Dec 22, 2011 11:50 pm

Use at least two (2) filesystem images: image 1 for the FAT partition, image 2 for the ext4 partition.

Just pick a size "plenty big enough" for the FAT partition.

The ext4 partition image should be no larger than necessary.

The installer should create two partitions: partition 1 the size of the FAT partition. Partition 2 the rest of the SD. Then DD both images to their respective partitions.

The ext4 partition will not be full. That is fine. Linux can resize at first boot. No tools needed.

The FAT partition will be full. It will not need resized because you defined the size when you created the image and the partitioning created the partition to be that same size.

bradburts
Posts: 341
Joined: Sun Oct 02, 2011 7:07 am

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sat Dec 24, 2011 10:11 am

Adding a subscription to this post.

I have had problems with Gparted & LVM in the past.
We should use LVM which would be a lot easier for newbies than migrating directories and symlinking. Less chance of killing it anyway.
Is there a robust linux command line partitioner that does it all?

@Chris
This topic is being discussed at http://www.raspberrypi.org/for.....038;t=1595 with a view to code an application.

sylvan
Posts: 118
Joined: Sun Nov 27, 2011 8:39 pm

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sat Dec 24, 2011 5:40 pm

gparted works fine but is rather complex.  parted (the cmdline version) is similar.  Only reason to use either is if you need to do something 'sfdisk' cannot such as GPT instead of MBR.

Forget about LVM.  It is just additional overhead during operation and more complexity during setup.  I don't know what you expect to get from it.

Use 'sfdisk' to partition.  It is easily scriptable.  You need two partitions.

Dump FAT image to partition 1.

Dump ext4 image (or other native linux filesystem, you don't care as it is an image) to partition 2.

Done.

edit:  new member?!?  bah!

MalayaTiger
Posts: 2
Joined: Sat Dec 24, 2011 11:19 am

Re: Help Wanted (Python): Adapt liveusb-creator to work with SD cards

Sat Dec 24, 2011 7:18 pm

So the step by step to use liveusb creator to make R-pi os is like this

First download the R-pi os distribution from raspberrypi.org

Second run the liveusb creator untill the process is complete (i want to use my camera as as memory card reader so the computer would detect it as usb storage)

Lastly run the os on R-pi !!!!!! :)

if one of this step incorrect please correct it
( sorry for bad english )

Return to “Python”