HISTORY: The start of raspbian (Debian Hard Float (armhf) for RPi)

[jamesh]

mpthompson said:


Abishur said:


I don't suppose while we're tweaking things, we could make it so that Debian doesn't stall for a long time when loading the R-Pi when it's not connected to a network?  Probably the wrong place to bring this up, but I thought I'd throw it out there for what it's worth


As it seems the Foundation will have a new Debian armel image out soon, you may want to send them suggestion to look into that issue.  Since I and 99.999% of everyone else here doesn't have any real hardware we have no way off addressing the issue.  In any case, we will be using the same exact kernel that Debian armel is using.


This is fixed in the next release. Also some other issues with Mac addresses being stored (so an SD card from one Pi has a 'hardcoded' Mac address which goes pear shaped when used on another Pi - or something like that). There's also a fix to set the clock on startup to the date of the latest log entry. Won't be accurate, but at least will be incrementing. Sure there is a load of other stuff as well. Was talking to the guy doing it all, he's been looking at wireless stuff, but that's difficult to make generic enough because of all the different firmwares required for different adapters.

[abishur]

Awesome!  Thanks James!

[mpthompson]

plugwash said:


I beleive the most debian-like way to run a repo like this would be to import the source first. Then wanna-build and buildd can pick up packages that haven't been built and build them.


Ah, that makes sense.  I'll look at it from the code being in the repository before the matching binaries are added.  That is probably why my searching failed to find scripts that handle it from the other side of of things.  I'll look into what wanna-build and buildd do.

BTW, I did manage to import the source for all the packages I've built so far into my new reprepro based repository.  Pretty simple actually.  For some reason I thought that source and binaries had to be added at the same time using .changes files and I missed that source can be imported by itself using the .dsc file.  That is why I was looking at things a bit backwards.  I'm assuming that source and binaries are really handled as two different entities by reprepro even though they share common directories under the source package name in the pool.

Something I'll look into is figuring out how to import all the source code that the standard armhf builds against into my repository and then updating it regularly.  Perhaps that makes more sense than bringing it all in piecemeal.  Reprepro has a mirroring features that may be just the ticket for doing this.

[mpthompson]

JamesH said:

This is fixed in the next release. Also some other issues with Mac addresses being stored (so an SD card from one Pi has a 'hardcoded' Mac address which goes pear shaped when used on another Pi - or something like that). There's also a fix to set the clock on startup to the date of the latest log entry. Won't be accurate, but at least will be incrementing. Sure there is a load of other stuff as well. Was talking to the guy doing it all, he's been looking at wireless stuff, but that's difficult to make generic enough because of all the different firmwares required for different adapters.
James, assuming that we make substantial progress on creating an Raspberry Pi flavored version of armhf over the next weeks/months, do you think there be much difficulty in getting the proprietary binaries recompiled for ARMv6+VFP and the hard float ABI?  It wasn't clear to me if these exist already or not.  If they don't the process shouldn't be much more than a simple recompile.

If I could get an answer to the question above, it would remove one of the unknowns that could shoot these efforts down if the proprietary binaries for the hard float EABI wouldn't be available.

[asb]

mpthompson said:


James, assuming that we make substantial progress on creating an Raspberry Pi flavored version of armhf over the next weeks/months, do you think there be much difficulty in getting the proprietary binaries recompiled for ARMv6+VFP and the hard float ABI?  It wasn't clear to me if these exist already or not.  If they don't the process shouldn't be much more than a simple recompile.

If I could get an answer to the question above, it would remove one of the unknowns that could shoot these efforts down if the proprietary binaries for the hard float EABI wouldn't be available.


Dom has already built hardfp libs and they're at https://github.com/raspberrypi/firmware

[mpthompson]

Quick update on things.  I made substantial progress in getting a new local repository created that will be the basis of the public repository that will be made available to everyone next week.  I'm also getting my head around the signing keys and such and how they are used in various parts of the build process.

I'm still on track for making the Raspberry Pi flavored Debian armhf repository publicly available on a hosted server next week..

On the package building side, I finally had some time to chase down why my builds of python2.7 were becoming infected with ARMv7 code.  The issue that was confusing me was that the python2.7 code includes source for libffi.a with one of its modules, but it turns out the package building uses the Debian installed version of libffi.a to statically link against and ignores the version of the same library in the python source code.  It took me a while to understand this was occurring.  It's one of those things that just took a few hours to peel down through the various layers of the build process to understand what was failing.  Rebuilding the libffi.a package solved the issue.  I'll be on the lookout for the same type of thing in the future when other instances of ARMv7 contamination occurs in packages.

[gimliflea]

You are doing a great service to the community. Thanks.

[mpthompson]

Had time this morning to register the following domains:

raspbian.com

raspbian.org

raspbian.net

These won't be setup until I get a hosting provider, but these will be where the public repository for the Raspberry Pi ARMv6+VFP Debian armhf port will be found.

Also, keep in mind that this project will only be useful until the Raspberry Pi Foundation releases a version of the Raspberry Pi that utilizes an ARMv7 capable CPU.  It's unknown when that will occur -- could be 6 months from now, could be years.  However, when it does the community can switch to using the standard Debian armhf port and the universe will once again be in balance and harmony.  Until then, Raspbian will be there -- hopefully .

[mpthompson]

Can anyone recommend a hosting provider that would provide reasonably priced hosting services for a Raspberry Pi flavored Debian armhf release?  Hopefully one that is very friendly to these types of projects.  My needs are for a dedicated  Debian system or Debian VPS (Virtual Private Server) with 200GB of disk space and as much monthly transfer allowance as possible.  The system would host a public mirror of the local repository I'm building into and some web pages to document how to use the repository. Because the recurring expense would be coming of my pocket I would like to keep the costs under $30 if possible, although I realize that may be unlikely.

[mpthompson]

Lots of frustration today trying to figure out how the various pieces of the Debian auto-builder system fit together. Unfortunately, these components are very poorly documented compared to the more familiar applications running on Debian.  For the most part, I have to put together information from bits and pieces in email list archives and from various web pages were people documented getting parts of an auto-builder system running. Often the information is old and outdated or conflicts with information found elsewhere.

There are three main components: sbuild, buildd and wanna-build.  I have sbuild, the part that actually builds a package from source files, working well.  Buildd, the part that coordinates package building and download/upload of packages from/to the repository, looks to be mostly working.  However, I won't really know for sure until the whole auto-build system is functioning.  The last piece, wanna-build, is proving to be the most difficult and least documented.  It's a wrapper around a SQL database that coordinates multiple build systems so that packages are build across the build cluster as efficiently as possible.  There are also various helper utilities such as quinn-diff that perform small, but essential, functions that need to be understood as well.

Despite the difficulties, it is vital that I get these tools functioning together so I can start cranking out RPi flavored hard float packages.  Hopefully one or two more days will be sufficient to get an autobuilder system in place and functioning.

In addition, I'm still also working on some ideas on how to host the public repository.  I'll probably have a temporary system set up next week just to get the packages available to people, but as the repository slowly matures as more and more packages are added, I'll then need to find a longer term solution.

[kasperl]

mpthompson said:


Can anyone recommend a hosting provider that would provide reasonably priced hosting services for a Raspberry Pi flavored Debian armhf release?  Hopefully one that is very friendly to these types of projects.  My needs are for a dedicated  Debian system or Debian VPS (Virtual Private Server) with 200GB of disk space and as much monthly transfer allowance as possible.  The system would host a public mirror of the local repository I'm building into and some web pages to document how to use the repository. Because the recurring expense would be coming of my pocket I would like to keep the costs under $30 if possible, although I realize that may be unlikely.



You could ask the guys hosting this place. There site looked more than capable. However, 30USD a month isn't going to buy you a lot of transfer, so it'd have to be a pretty charitable thing.

Oh, and the Python 2.7 thing sounds like the kind of bug the Debian maintainer would like to know about if it isn't in Debian BTS already.

Thanks for all the hard work BTW, I'll definitely be using this to power my Pi once I get one.

[plugwash]

Kasperl said:


You could ask the guys hosting this place. There site looked more than capable. However, 30USD a month isn't going to buy you a lot of transfer, so it'd have to be a pretty charitable thing.


I've looked at mythic beast's price lists and unless they are prepared to do this on a purely charitable basis their prices for bandwidth make them a non-starter for this kind of project. Similarly priced packages from leaseweb offer an order of magnitude more transfer.

[mpthompson]

Kasperl said:

Oh, and the Python 2.7 thing sounds like the kind of bug the Debian maintainer would like to know about if it isn't in Debian BTS already.
Thanks for all the hard work BTW, I'll definitely be using this to power my Pi once I get one.


This issue with Python wasn't really a bug, but rather more simply the way the maintainer chose to configure the build of Python.  I'm assuming that there are probably guidelines that when an application includes versions of libraries that already have a Debian package, choose to use the Debian package instead.  These are usually configure settings specified when the package is built.  It just took me a little while to understand that is what was going on and why I was getting a static link against code that hadn't yet been optimized for the RPi.

[mpthompson]

Kasperl said:


You could ask the guys hosting this place. There site looked more than capable. However, 30USD a month isn't going to buy you a lot of transfer, so it'd have to be a pretty charitable thing.


Good idea.  Does anyone have contacts with the person who would be best to ask about such a favor?  As far as I can tell, pretty much everything involving Linux is a charitable thing and mostly a labor of love.  Fortunately, there seems to be enough good will to keep the entire Linux community going.

[Update: Oops, I see plugwash already identified mythic beast as the hosting provider and looked into their prices.  I agree, leaseweb seems to off much more transfer capability for a relatively modest price.  Perhaps someone from the RPi Foundation could chime in whether approaching them on a charitable basis might work.]

[tufty]

I"d suggest going through a repository provider like github, gitorious, or code.google.com, all of which are free for open souce projects.

Simon

[mpthompson]

tufty said:


I"d suggest going through a repository provider like github, gitorious, or code.google.com, all of which are free for open souce projects.

Simon


I believe those are Git and SVN repositories rather than Debian package repositories and thus unsuitable.  Please correct me if I'm wrong.

I did see that someone came up with a way of hosting a small Debian repository on Google's app engine.  If it could be done with a few packages, could it be done with 10,000's?  Hmmm.

http://jyro.blogspot.com/2009/.....ngine.html

[Blars]

I'm using ramhost.us for my personal virtual server, the main problem is they tend to be sold out most of the time.  (they have server farms in the uk and germany as well as the us)  You will obviously need more disk space and bandwidth than my $5/month account gives me.

[plugwash]

Blars said:


You will obviously need more disk space and bandwidth than my $5/month account gives me.


Looking through their packages they don't seem to have much to offer that would suit a project like this. Even on their top packages bandwidth and diskspace are just not high enough IMO.

It won't reach that size immediately but in the long term (after the etch release when we have both stable and testing in the repo) we are likely to need over 100GB of disk space for the repositry itself and to allow for things like making backups before making risky changes IMO we really don't want the repositry to be bigger than half our total available disk space.

That is the problem really, debian is HUGE and it's not until you try and setup a project like this that you realise just how huge it is.

[shirro]

How about this?

Find your closest Debian mirror: http://www.debian.org/mirror/list

Find out which College/University/ISP that is hosting it. How about your old college? Have a chat to their IT people and see it you can add it to there existing terabytes of storage and massive bandwidth. Go from there.

If I was doing this I might talk to my ISP (though they only do local mirrors), then put out a call for help on the lug mailing list and on the national linux mailing list. I know there are people who run mirrors on there as well as staff at universities and the national academic research network.

[mpthompson]

shirro said:


Find your closest Debian mirror: http://www.debian.org/mirror/list

Find out which College/University/ISP that is hosting it. How about your old college? Have a chat to their IT people and see it you can add it to there existing terabytes of storage and massive bandwidth. Go from there.

If I was doing this I might talk to my ISP (though they only do local mirrors), then put out a call for help on the lug mailing list and on the national linux mailing list. I know there are people who run mirrors on there as well as staff at universities and the national academic research network.


Shirro, all good ideas.  I've reached out to debian-ports.org as that is basically the site that handles unofficial ports of Debian.  We'll see if my email garners a response over the next few days.  I'll also start reaching out to other mirroring sites and see what I can turn up.

[mpthompson]

A status update.  I've made pretty good progress on autobuilding.  I haven't yet dotted the i's or the crossed the t's, but I have the major pieces in place and somewhat working through the following loop:

repository->wanna-build->buildd->sbuild->dupload->repository

It's going to take a few more days to get all the scripts working together and such.  Also, parts of the connections between a few of the steps involves email which is a whole other adventure.  I either need to get sendmail set up on my home Linux systems or I short circuit that path in the process.  I also need to insert the script to filter out ARMv7 contamination as I'm still relying on regular armhf packages to satisfy build dependencies.

Once I have confidence that the autobuild scripts don't need additional debugging, I'll begin the import into the repository of all Debian source and 'all' binary packages.  While the repository is fairly small (a few hundred packages) the autobuild scripts are much easier to debug as the data set is small and the internal helper databases can be quickly rebuilt as needed.  Once the import occurs, I can sever the link to the ARMv7 armhf packages and have all dependencies built automatically.  However, the local repository will instantly grow from under a gigabyte in size to about 100 gigabytes which is what makes hosting it a bit painful.

As the automatic builds become routine, I'll look to augment the single iMX53 QSB doing the building with at least four additional iMX53 QSB systems as plugwash suggested.  This will help speed up the building process significantly.  As the project shows momentum and people can download the packages being built, perhaps I can then solicit donations to purchase even additional infrastructure and pay for real hosting.

As I've been putting the majority of my time in getting autobuilding going, I haven't had too much time to spend on looking for hosting solutions.  I'm leaning towards getting a minimal quick & dirty solution in place that won't last more than a few months and worry about a longer term robust hosting solution down the line.  If I go the quick & dirty route, I'm still looking to have something out by the end of next week that people can begin using.

Finally, I'll be looking to reconfigure a portion of my home network on an independent, secure vlan.  This will allow me to isolate the systems devoted building the packages and holding the local repository so I can then open up that portion of the network to select volunteers that can help me manage the build servers.

Lots to do, but progress is being made.

[shirro]

Are you going to wait until you can get a Pi and run some benchmarks before you order more imx53?

I say that because I grabbed a genesi efikamx debian hf download and loaded it up on my imx and did some rough X benchmarks and it was quite a but slower than my usual setup. Different CPU to the pi ofcourse and the people who put that efika hf stuff together claim it is much faster. Also I may have had something configured wrong.

[mdewey]

On the hosting issue I seem to remember that the Foundation members have connections with the University of Cambridge. UoC does not appear on the list of Debian mirrors AFAICS but perhaps they might be worth approaching via the Foundation?

[plugwash]

Just thought i'd tell you that while browsing arround on leasewebs site they currently have a special on an unmetered 100 megabit server at E69/month (lower than the E99/month I mentioned previously).

[mpthompson]

Was just looking at Leaseweb and I had to make a selection to see the price.  Hmmm, about $90 a month.  Tempting.  I'll have to look at the terms of service and make sure that isn't just for the first month.  Also, I'll need to make sure this a monthly contract that I can cancel at any time.  I don't want to be stuck with a long term contract if this project fails to happen for some reason (i.e. the RPi Foundation announcing a Cortex A8 based RPi next month) or an official Debian port is announced that will support the RPi.

Guess I better talk to my wife about setting a budget around this project.  Should be an interesting conversation.