The start of raspbian (Debian Hard Float (armhf) for RPi)


 
614 posts   Page 23 of 25   1 ... 20, 21, 22, 23, 24, 25
by hexelpdkk » Mon May 28, 2012 10:03 pm
And some more kernel packaging bits - I finally found a site with an explanation of "the debian way" of packaging kernels - http://users.wowway.com/~zlinuxman/Kernel.htm - which seems up to date. After a first pass through this, I think my next step is getting everything going using make-kpkg and looking at putting a boot config file modification into /etc/kernel/postinst.d. If I use latest firmware, and with the bootc patch applied to the kernel, this can just be the normal vmlinuz kernel.
User avatar
Posts: 128
Joined: Fri Feb 24, 2012 4:40 pm
by plugwash » Mon May 28, 2012 11:23 pm
No, those are instructions for building a custom kernel for your own use. For raspbian we need ot modify the debian source package that builds the standard debian kernels to build the kernels we want.
Forum Moderator
Forum Moderator
Posts: 2176
Joined: Wed Dec 28, 2011 11:45 pm
by mpthompson » Wed May 30, 2012 5:56 am
Build update for Tuesday evening. The binary packages count is now:

Debian armhf 34437
Raspbian armhf 33853

Just under 600 packages left to build before parity is reached with Debian armhf. The counts are going a little slower because we've ran into some difficulties with updates from Debian Wheezy that temporarily gave the autobuild systems some fits. Also, we seem to be getting a lot of updates from Debian Wheezy which the autobuilders then have to rebuild to bring the Raspbian packages up to date. This rebuilding cuts into the resources available to compile new packages. I suspect the updates are more frequent as Debian is looking to freeze Wheezy into the next stable release in late June or July. Many developers are trying to get their packages into the Debian Wheezy before the freeze occurs.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by mpthompson » Sat Jun 02, 2012 7:53 pm
The build count for Saturday afternoon is:

Debian armhf 34644
Raspbian armhf 34320

Something that occurred today is that the build queue has now finally been depleted. All the packages that can be built, have been. However, there are about 160 packages where the builds failed and these packages in turn are keeping other packages from building because of dependencies. So to close the gap between Debian Wheezy armhf and and Raspbian armhf we'll need to carefully start prioritizing and examining the issues that are preventing these remaining packages from building.

Also, the build systems will continue to be busy as we update each day the new packages that are being updated in Debian Wheezy. At this time, usually 60 to 90 source packages are updated each day which then need to be rebuilt in Raspbian. Once Debian Wheezy freezes, this daily update of packages should be much less.

Finally, there are issues with the edos-debcheck application that autobuilders use to figure out the dependencies between what binary packages are available for installation and the source packages that are dependent upon those binary packages for building purposes. Plugwash has been after me to look into the issue, but I haven't had much of an opportunity the last few days. Hopefully once that issue is fixed dozens of more unbuilt packages will be built.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by john.mills » Sun Jun 03, 2012 5:35 am
HI MIke,

Now that Debian main is almost completed porting across will you be turning your attention to porting the Debian non-free repositories? If some of the build machines are available then this would be a very good idea. There are a lot of packages that would be really useful to have available on the Pi and provide a much better user experience going forward.

Best regards,

John
Posts: 81
Joined: Mon Apr 09, 2012 5:23 am
by KeithSloan » Sun Jun 03, 2012 5:46 pm
Congrates on making it so close to the finishing line. Look forward to the news when you finally cross the line. Yours is a magnificent effort !!!
Posts: 162
Joined: Tue Dec 27, 2011 9:09 pm
by mpthompson » Sun Jun 03, 2012 7:46 pm
john.mills wrote:Now that Debian main is almost completed porting across will you be turning your attention to porting the Debian non-free repositories? If some of the build machines are available then this would be a very good idea. There are a lot of packages that would be really useful to have available on the Pi and provide a much better user experience going forward.


We'll be looking into this soon. I'll need plugwash's help as he's much more familiar with the ins & outs of the Debian repository and the configuration changes on our side to accommodate multiple repositories in our infrastructure. At a minimum, we'll get the 'all' architecture packages which include a lot of USB device firmware which will make many USB devices usable under Raspbian (ie. USB wifi dongles and such). How soon we'll be able to devote the build systems to the actual architecture dependent packages is more of an open question. I do have additional hardware on order, but it won't get here until the last week of June at the earliest.

One of the tricky things is we have to make the configuration changes on an infrastructure that now exceeds 80GB of compiled armv6 code. Although it's backed up, I still want to tread very lightly and make sure we don't break things that could take days, or longer, to fix.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by ian.macarthur123 » Sun Jun 03, 2012 8:33 pm
mpthompson wrote:I do have additional hardware on order, but it won't get here until the last week of June at the earliest.


So, "additional hardware" - a Beowulf cluster built out of hundreds of little R-Pi's...!?

Well, maybe not.

(Sorry, just joking, but...)
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by mpthompson » Sun Jun 03, 2012 9:38 pm
Actually, we use Freescale iMX53 Quick Start Boards to build Raspbian. These are 1GHz, 1GB, ARM devices with SATA interfaces. The largest Debian packages barely build on these systems. Anything less and Raspbian would be largely impossible to build.

I have three additional with SATA drives on order to supplement the five already used to build Raspbian. Unfortunately, these devices seem to be in short supply.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by ian.macarthur123 » Sun Jun 03, 2012 10:00 pm
OK - those look quite neat, actually... Reminds me of my old beagle, only, newer...

To be honest, I'd had envisaged a more "conventional" cross-compilation environment - by which I mean a number of x86 boxes cranking out cross-compiled arm binaries (which is a part of what I do at work some days...) rather than using these, which are almost-but-not-quite native builds, I guess.

I imagine that makes a lot of things easier for configure, though must leave you with occasional really subtle host/target differences to resolve.

I know there are a lot of packages out in the wild that will not build at all, or at least not without a lot of manual intervention, in my "traditional" cross-compilation environment, so this way seems like a win in that regard!

Must be some costs associated with maintaining the build-farm though - you ok for funding on this?
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by plugwash » Sun Jun 03, 2012 11:43 pm
ian.macarthur123 wrote:OK - those look quite neat, actually... Reminds me of my old beagle, only, newer...

CPU wise and price wise they are comparable to the beagleXM but unlike the beagleXM they have 1GB of ram, SATA and native ethernet (the ethernet on the beagleXM is USB based).

which are almost-but-not-quite native builds, I guess.

They are native builds. Everything inside the build chroot on the autobuilers is from raspbian.

I imagine that makes a lot of things easier for configure, though must leave you with occasional really subtle host/target differences to resolve.

It's not generally a huge issue. Occasionally a package makes inappropriate assumptions but that can happen on any architecture. Having a build machines support more features than the port requires is not exactly unusual.

Must be some costs associated with maintaining the build-farm though -

There are obviously capital costs in building and expanding the farm but on an ongoing basis it doesn't need much, just a bit of power (indeed I bet that the repo host probablly draws more than all the autobuilders put together :/)

Currently the biggest ongoing cost by far is paying for the public repo server. To avoid really nasty surprises we needed a host with unlimited traffic and that is relatively expensive.

you ok for funding on this?

We do plan to set up a donations page soon.
Forum Moderator
Forum Moderator
Posts: 2176
Joined: Wed Dec 28, 2011 11:45 pm
by Montala » Mon Jun 04, 2012 12:29 pm
plugwash wrote:We do plan to set up a donations page soon.

Please do... I am sure that there are many members on here (including myself) who would be more than willing to make a donation to this project... especially if it would help to move things along at all!
User avatar
Posts: 639
Joined: Mon Mar 05, 2012 11:54 pm
Location: Herefordshire (U.K.)
by Baranyk » Wed Jun 06, 2012 10:27 pm
Registered on the Wiki as DanielFenton. If you're okay with my help, I'm ready to help document.
Posts: 7
Joined: Sat Jun 02, 2012 6:32 pm
Location: Pennsylvania, USA
by mpthompson » Wed Jun 06, 2012 11:29 pm
Baranyk wrote:Registered on the Wiki as DanielFenton. If you're okay with my help, I'm ready to help document.


Terrific. I added you as an editor on Wiki. Document away...
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by mfc_alpha » Mon Jun 11, 2012 12:11 pm
Hello,

Just 2 questions, but I'm not sure that they are relevant for raspberry.

First, Linaro get some performance improvement using last gcc version (http://www.cnx-software.com/2012/06/03/ ... -omap4430/) .
Could we benefit from this ?

The Second one is about the build system. Is it possible to benefit from emulation on x86 ?
I was used to build arm binaries using scratchbox and I just find this (http://www.stgraber.org/2012/02/03/ever ... u-precise/). This was just for simple binaries not the whole system.

Thanks !
Posts: 6
Joined: Mon Jun 11, 2012 10:07 am
by plugwash » Mon Jun 11, 2012 1:40 pm
mfc_alpha wrote:First, Linaro get some performance improvement using last gcc version (http://www.cnx-software.com/2012/06/03/ ... -omap4430/) .
Could we benefit from this ?

Replacing gcc with a linaro version would be a risky move and not one I plan to make. Hopefully in time improvements from gcc will filter up to upstream gcc and then down to the debian gcc packages. Anything recent will probablly have to wait for wheezy+1 though.

Optimisation levels are generally at the packagers discretion, most debian packages use -O2 but if you beleive there are cases where -O3 is appropriate and isn't being used please take that up with the debian maintainers of said package in the first instance. In extreme cases where a significant gain can be found from -O3 and the debian maintainer is uncooperative we may do this ourselves but I'd rather avoid it if possible.

The Second one is about the build system. Is it possible to benefit from emulation on x86 ?
I was used to build arm binaries using scratchbox and I just find this (http://www.stgraber.org/2012/02/03/ever ... u-precise/). This was just for simple binaries not the whole system.

I set up a vm with qemu-user chroots and tried building apt in it, the result was that the build took slightly longer than it did on an IMX board (admittedly the host was only a pentium G).

The main case where emulation would seem useful is building stuff that needs massive ammounts of ram. Currently some packages cause our autobuilers to swap for days.

I have not investigated cross compilers yet, it is probablly possible to build them from our gcc source packages but all the debian documentation on doing so seems to be out of date and so I don't know how many problems you would run into. Furthermore debian packages are not required to support cross-compilng so this is unlikely to be an option for autobuilding.
Forum Moderator
Forum Moderator
Posts: 2176
Joined: Wed Dec 28, 2011 11:45 pm
by mfc_alpha » Mon Jun 11, 2012 2:04 pm
OK, Thanks for your explanations !
Posts: 6
Joined: Mon Jun 11, 2012 10:07 am
by mpthompson » Mon Jun 11, 2012 7:11 pm
Time for another package update. The current binary package counts now stand at:

Debian armhf 34816
Raspbian armhf 34737

As of June 11th, Raspbian is hovering at just under 80 binary packages under Debian Wheezy. Out of 34,816 packages, a gap of 80 packages is not bad at all. Quite frankly, I'm surprised we have been able to get this close. I'll have to check to make sure there are not orphaned binary package in the Raspbian repository that may be artificially increasing the Raspbian package count.

I'm not certain we'll narrow this gap by much as many of these packages have build problems that are unlikely to be resolved, unless someone really needs one of the packages in question to be built and has the time to sort through what the issues might be.

An up to date list of the packages that aren't building in Raspbian can be found here:

http://www.raspbian.org/BuildAttempted#preview
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by plugwash » Mon Jun 11, 2012 7:27 pm
mpthompson wrote:As of June 11th, Raspbian is hovering at just under 80 binary packages under Debian Wheezy. Out of 34,816 packages, a gap of 80 packages is not bad at all. Quite frankly, I'm surprised we have been able to get this close. I'll have to check to make sure there are not orphaned binary package in the Raspbian repository that may be artificially increasing the Raspbian package count.

Note that due to the way debian testing works it is likely that we have some stuff that debian wheezy armhf does not. In particular I beleive this applies to haskell packages since there has been a haskell transition going on for a LONG time.
Forum Moderator
Forum Moderator
Posts: 2176
Joined: Wed Dec 28, 2011 11:45 pm
by Zinahe » Thu Jun 14, 2012 5:11 am
Damn ! It's 1:08 AM in the morning and it took me more than 2 and half hours to read through all the 23 pages of this post. Big ups to plugwash and mpthompson for such an amazing contribution to the RPi community. I've sent $25 your way as a small token of my gratitude.

Keep up the good work.
Posts: 16
Joined: Sun Jan 29, 2012 11:56 pm
by mpthompson » Thu Jun 14, 2012 5:59 am
Zinahe wrote:Damn ! It's 1:08 AM in the morning and it took me more than 2 and half hours to read through all the 23 pages of this post. Big ups to plugwash and mpthompson for such an amazing contribution to the RPi community. I've sent $25 your way as a small token of my gratitude.

Keep up the good work.


Thank you very much for the donation. We'll do our best to keep things moving forward.

Today I found myself reading the first few pages of this thread so I could understand how I did some of the things early on. Amazing what is forgotten in just a few months time.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by MisterAlex » Fri Jun 15, 2012 8:38 pm
Is there a Twitter feed I could follow to keep abreast of Raspbian progress?
User avatar
Posts: 7
Joined: Wed Jun 13, 2012 1:14 pm
Location: Dallas
by mpthompson » Fri Jun 15, 2012 8:47 pm
MisterAlex wrote:Is there a Twitter feed I could follow to keep abreast of Raspbian progress?


There will be. I just haven't got around to setting it up just yet. This will be a reminder for me to take care of it.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by mpthompson » Fri Jun 15, 2012 8:51 pm
Time for another package update. Something interesting has happened:

Debian armhf 34816
Raspbian armhf 35013

Raspbian now has more binary packages than Debian Wheezy armhf. Hmmm. Plugwash indicated this might happen and I'll have to get his insight into this.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by hadi » Wed Jun 20, 2012 6:26 pm
is there anyway to use armel packages?

I don't exactly know what is the difference between armhf and armel.
Posts: 29
Joined: Fri Jun 15, 2012 4:28 pm