User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Wed Apr 25, 2012 7:27 pm

Last night was the first time I got all build servers plugged in and going at once.  A picture of the cluster as it looks now in its nice, new, handcrafted rack can be found here:

http://home.comcast.net/~micha.....r_rack.jpg

OK, the rack looks a little hokey, but $25 for wood and screws, it beats what a real rack would cost.

If you examine the photo you'll notice that there are three empty slots.  Hint, hint, hint… I would gladly accept donations of matching iMX53 QSB and SATA drives to complete the cluster of eight build servers.  Once the major build push is done in a few months I'll send you the iMX53 QSB and hard drive back to you if you wanted it for your own use.  My cost for a iMX53 QSB and hard drive is about $230 after taxes and shipping.

Having them all working I couldn't resist running a test overnight.  After nine hours, the cluster built 285 source packages yielding 360 binary packages to be added to the repository.

On a sad note, Mr. Murphy payed me a visit this morning.  I needed to adjust something with the the Build00 server and with power going to the cluster I unplugged it's molex connector and when plugging it back in I accidentally had the plug reversed.  The molex connectors won't plug in that way, but the pins stick out just far enough that 12V was momentarily connected the Build00 unit on it's 5v input.  Instantly the power supply made a popping noise and all the build servers instantly switched off.  Seeing what I had done I let out a big "OH! SH**!!!"  Dumb, dumb, dumb…

Fortunately, the 12v surge was isolated to just Build00 instantly frying the hard drive, but somehow the iMX53 QSB survived.  Being a development board it must have some type of protection against a klutz like me attaching it to 12v.  All the systems going down and the popping noise was just the PC power supply circuit protection kicking in.  A new Build00 hard drive is already on order with Amazon.

Unfortunately, Build00 was my prototype system and where I kept all my Linux kernel builds of which, of course, I hadn't yet made backups of.  I have notes of that work, but those may be incomplete and I'll probably be in for some heartache trying to replicate it as I sort out the kernel issues I came across late last week.  Losing my kernel work is not good, but I'm very happy I didn''t accidentally fry $1000 worth of ARM development boards and hard drives.

Needless to say, I clipped the 12v line going from the power supply to the molex connectors and I'll be much more careful about doing anything while power is applied to the cluster.  I'll also be spending some time putting a backup regime in place.

Oh well, it's been a two steps forward, one step backward morning for me.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 6:42 am

Not sure how many people are still following this thread and my adventures/misadventures in getting a version of Debian hard float going.  Despite frying a hard drive today I feel things are finally starting to come together and we should see very significant progress over the next few weeks with regards to this project.  By significant, I mean a threshold being reached of the repository containing an interesting number of packages.

Four of the build cluster servers are humming along (the fifth will be up in two days) and a few of the issues that have slowing me down have finally been resolved.  With some advice from Steve McIntyre, one of the leaders of the Debian armhf effort, I now have a stable and fast kernel running on the iMX53 build systems that is working with my SATA drives.   I also figured out some sbuild/schroot configuration issues that were causing about 10% to 20% of my builds to fail because of dependency issues.  Finally, or the most part, the package build process is automated. Some parts are still manual, but but I'll have them automated as soon as I have enough time to site down for a few hours and write some Perl scripts.

Another significant improvement will be plugwash volunteering to help start chasing the problematic packages that need TLC before they can be built in an automated manner.  Plugwash has been very helpful with giving me advice and I'm certain the two of us working together will iron out build issues much faster than I have been working alone.  He's been pretty patient while I've worked out issues of granting outside access to the build cluster, but those issues should be worked out within the next 24 hours.

The only big question I have now is whether I'll have actual RPi hardware before the repository is mostly complete or not.

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 8:25 am

This is a very noble effort.

But please could you explain to the uninitiated what sort of benefit you are hoping to get out of this. The reason I ask is that I find it hard to relate to an operating system having lots of floating point maths.

Will the recompilation also help non floating point maths in some way?
i.e. Compiled specifically for Pi processor?

Does X use floating point or integer maths?
What sort of percentage improvement are you expecting?
What sort of percentage improvement would you consider success?

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 12:48 pm

mpthompson said:


Not sure how many people are still following this thread and my adventures/misadventures in getting a version of Debian hard float going.


I am   Spent yesterday evening and this morning reading through this entire thread. As other posters have said the amount of enthusiasm and effort (and resources!) you're throwing at this is truly admirable. When my RPi eventually turns up, this is definitely something I'll have a play with. Are you still planning on releasing a Qemu image?

Good luck with future progress.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 4:15 pm

KeithSloan said:


But please could you explain to the uninitiated what sort of benefit you are hoping to get out of this. The reason I ask is that I find it hard to relate to an operating system having lots of floating point maths.


Well, in the case of the Linux kernel, no floating point math is used within the operating system at all.  In fact, the same exact kernel can be used for both the hard float version of Debian as well as the soft float version.

So why this project?  As is probably fairly obvious, where gains will be made are anywhere floating point calculations are performed.  This will come in two major ways.  First, the floating point hardware on the CPU will be utilized to it's fullest and code that executes a significant number of floating point calculations will benefit significantly with users perhaps experiencing a 10x or greater performance increase.  The other optimization is in the way floating point data is passed between function calls -- the application binary interface or ABI.  In a soft float ABI, floating point data is transferred between functions calls in the integer registers and must be loaded/unloaded to/from these registers anytime a function call is made involving floating point data.  With a hard float ABI, the CPUs floating point registers are utilized to transfer floating point data so there is an extra boost in performance that comes from not shuffling floating point data around.  This is boost is not as significant the use of hardware floating point instructions, but it is a measurable increase.


Will the recompilation also help non floating point maths in some way?
i.e. Compiled specifically for Pi processor?


Yes, non-floating point code should be helped, but the increase in efficiency will be marginal.  This will come because the packages are compiled using armv6 instructions rather than the more generic and less powerful armv4 instruction set used in the existing soft float Debian distribution.  I'm not very knowledgeable with regards to ARM instruction sets, but it's my understanding that ARM has made improvements between armv4 and armv6 in the instruction set with regards to more efficient access to memory and such.  Perhaps someone more knowledgeable about ARM instructions can fill us in what might be expected here.


Does X use floating point or integer maths?
What sort of percentage improvement are you expecting?
What sort of percentage improvement would you consider success?


These questions are hard to answer as I'm looking at success from a different perspective.  The Raspbian port of Debian armhf will be as optimized for the Raspberry Pi as one can reasonably make when looking at it as an entire Linux distribution.   If whatever gains are made are found useful and this distribution is used by a significant percentage of the Raspberry Pi community, then I'll deem it successful.

Once the repository is complete, the nice thing is that a Raspberry Pi user will be able to choose to use Debian Linux on their Raspberry Pi without feeling that in making the choice they are compromising on performance.  I believe that will be a great benefit to the Raspberry Pi community and to the Debian community and in itself justifies this project.  As no one seemed to be stepping up to the plate to make it happen and the Raspberry Pi Foundation was looking towards other favored Linux distributions, I saw an opportunity to contribute back to the both the Raspberry Pi and Debian community and ran with it.  Along the way I'm learning an incredible amount about Linux, Debian, and managing large build projects is a personal bonus as well.

For me personally, my hobby interests are in robotics and teaching my kids programming.  Both of these interests will be furthered by having a hard float version of Debian available for the Raspberry Pi, but on their own don't justify my efforts.  Rather, I'm really motivated by a desire to give back to the community and what I might learn along the way.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 4:29 pm

Andrew Scheller said:

Are you still planning on releasing a Qemu image?
Yes, that has been my intention, but up until now my efforts have been focused on getting the autobuilding system running so that there is an interesting number of packages to be found in the repository.  This took longer than I though (doesn't it always) and I haven't had a chance to put together a QEMU image.

As I'm just switching on the autobuilders now, hopefully things will settle down within a few days and I'll have a chance to work on a QEMU image.  My hope is that I can put together a QEMU image with enough on it that someone could download it and be able to use it to help fix packages that otherwise won't compile.  In addition to the build-essential tools which are already there, this will include all the Debian packaging tools, some of which I believe are still missing from the repository and may have to be handbuilt.

I'll try to have an update on this within a few days.

mdewey
Posts: 37
Joined: Wed Dec 07, 2011 10:47 am
Location: UK

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 5:12 pm

KeithSloan said:


This is a very noble effort.

But please could you explain to the uninitiated what sort of benefit you are hoping to get out of this. The reason I ask is that I find it hard to relate to an operating system having lots of floating point maths.

Will the recompilation also help non floating point maths in some way?
i.e. Compiled specifically for Pi processor?

Does X use floating point or integer maths?
What sort of percentage improvement are you expecting?
What sort of percentage improvement would you consider success?



Perhaps I could give an example of a use case which would benefit from this project. I use the open source statistical system R and occasionally run simulations which occupy one core of my fairly old desktop for tens of minutes with accompanying fan noise and so on. If I had a RasPi I could leave the simulations running silently and using much less power in the corner while I get on with something else. I have no idea how long such a simulation would run for but the sort of speed up which has been discussed earlier in this thread would make a huge difference to the running time and maybe make the difference between the project being viable or not. Only time will tell of course and I have not yet even ordered a RasPI ...

This is the most amazing thread on the site and I stand in awe of the people who understand it and are contributing to making my life easier.

Michael

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 6:56 pm

"I use the open source statistical system R"

But could you not just compile  the open source statistical system R with the floating point options? Why do you need the whole operating system as well. Did I miss something?

I wanted to run World Community Grid and [email protected] but given the device is only equivalent to a 300Mhz Pentium III its not really an option (unless Broadcom decide to open up the GPU)

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5370
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 7:18 pm

KeithSloan said:


"I use the open source statistical system R"

But could you not just compile  the open source statistical system R with the floating point options? Why do you need the whole operating system as well. Did I miss something?


Well it is quite likely that "open source statistical system R" depends on other packages. It almost certainly uses libm, and probably many other.

Something like midori depends on hundreds of packages. It is very hard to determine what packages need to be build with hardware floating point. Possibly midori will run significantly faster if freetype is built with floating point. Or maybe it's webkit, or jpeg, or crypto, or cairo, or xfonts...

If you build them all, then you what you've got is good.

tomtor
Posts: 44
Joined: Sun Apr 08, 2012 2:19 am
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 7:28 pm

KeithSloan said:


But could you not just compile  the open source statistical system R with the floating point options? Why do you need the whole operating system as well. Did I miss something?


You cannot just compile with the floating point options (-softfp) and have the same efficiency gain because all the libraries in the default Debian distribution expect floating point parameters to be passed in normal registers.

Because of this, calls which pass floating parameters between functions in a program like R are also inefficient. So the floating point instructions can only be used efficiently within the same function.

Partial solutions exist for this problem (like -flto link time optimization which effectively inlines most function calls), but those are difficult to use and you can still not use libraries in an optimal manner.

You are right that the difference will be small for many applications, but for those applications which are floating point intensive, the gain will be big.

plugwash
Forum Moderator
Forum Moderator
Posts: 3475
Joined: Wed Dec 28, 2011 11:45 pm

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 7:47 pm

mdewey said:

 I use the open source statistical system R
Be aware that debian has recently been having problems building many R packages for arm. If you are interested in R on arm going forward then it might be an idea to get involved.

http://lists.debian.org/debian.....00015.html

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 9:01 pm

Been reading about Debian armhf effort for later versions of ARM. ( Later tha Pi)
got directed to http://www.powerdeveloper.org/.....efeb6aeab7

Towards the end it goes into detail about problems with Open/GL and not able to exploit the GPU as the blob has to be statically linked and this is not possible when using armhf is trying to link in a blob that is not.

Would that not also apply for the Pi given that Broadcom supply the blob of Open/GL and it may not have an armhf version

plugwash
Forum Moderator
Forum Moderator
Posts: 3475
Joined: Wed Dec 28, 2011 11:45 pm

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 9:05 pm

I believe dom has already built hardfloat versions of the gl drivers etc. I dunno if anyone has tested if they actually work or not though.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 10:27 pm

When the time comes, I have faith that the Foundation will help make everything available for the Debian armel available for Raspberry Pi flavored Debian armhf.  They seem to have deep connections in Broadcom necessary to make this possible.  I may be naive about this, but it seems illogical for them not to. By that point it should just be a simple install of the distribution and recompile of existing packages (I'll help them do it if they pay my air fare .

Given the longer history that Debian has on ARM devices and some of the criticisms I hear regarding Fedora on ARM (I really can't judge the criticisms myself), I really hope that this distribution of Linux becomes the primary distribution of Linux for the Pi.  Perhaps the Foundation will even reconsider using Debian as their primary Linux of choice as I believe it has specific advantages in the education market.

It's still far too premature to push for any of this, but that is my hope.  The best way I can make the case is to move this distribution as far forward as possible and hope that eventually it will gain a momentum of it's own through community efforts.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Thu Apr 26, 2012 11:01 pm

My daily update.  For the last 22 hours I've had four build servers autobuilding packages.  The total over this time is 1013 software packages built with 1272 binary packages moved into the repository.  Now, these are the low level, smaller packages with very few build dependencies, but it's still pretty good progress and a good warmup for the build marathon over the coming few weeks.

There were also about 128 packages that failed autobuilding for one reason or another.  Unfortunately, I don't have time chase down why these packages are failing to autobuild.  Right now, I'm concentrating instead on the low hanging fruit which is the 90% of packages which will build without issues.  For the others I'll have to manually go through them to determine they do in fact build with the official Debian armhf and if so, why are they failing to build for me.  Hopefully most of them have a common cause that can be fixed in the build environment so they don't necessitate changes to the package source itself.  We'll see.

Also, I received the replacement hard drive from Amazon and I now have five autobuilders at my disposal.  Normally, four of the builders will be autobuilding and an extra one will be reserved for tracking down why some packages are having problems.

At this rate, I feel that it shouldn't take much more than a month to flesh out most of the Raspberry Pi flavored Debian armhf repository.    However, I'm not counting time it may take to track down issues as they crop up.

wcchandler
Posts: 6
Joined: Sat Oct 01, 2011 1:29 am

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 12:20 am

I built a chroot of /wheezy to mess around with.  I know you have a nice build environment/setup, but I would like to help out if possible.  Do you have a list of packages that have yet to be compiled?  Might be fairly trivial to set it up on raspbian.  I completely understand if you want to do it all by yourself and just need someone to guinea-pig it out.  Not to mention your setup might not be conducive to outside help.

Either way let me know!

joemegahertz
Posts: 11
Joined: Wed Nov 16, 2011 9:48 am

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 2:23 am

Once again thanx for all the hard work.  Has anyone actual tried this on the pi yet?  Only reason I ask is you have put a lot of time and money into this and I'd hate to see some option or something complicated overlooked.  All this work and doesn't boot properly, ouch! Also it sounds like you have the most understanding wife on the planet.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 3:16 am

joemegahertz said:


Once again thanx for all the hard work.  Has anyone actual tried this on the pi yet?  Only reason I ask is you have put a lot of time and money into this and I'd hate to see some option or something complicated overlooked.  All this work and doesn't boot properly, ouch! Also it sounds like you have the most understanding wife on the planet.


Yes, Dom did boot one of the early images I created on actual Pi hardware so the packages to indeed work on real hardware.  Also, I believe a few others may have tried it and ran some performance tests.  Finally, I am running the package on real ARM hardware so I know they work, but just not the actual Pi hardware.

And yes, I do have a very understanding wife.

veryevil
Posts: 30
Joined: Fri Mar 09, 2012 3:38 pm

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 11:16 am

Hi, I just heard from farnell that my second Raspberry PI will be delivered on the 7/5/2012 and as I will then have two I would like to loan it out to you while you wait for yours.

I think the work you are doing here is spectacular and I thoroughly support it. I hope that having a PI sooner rather than later will help speed thinks up and help you out.

When you get the one you have ordered you can simply post it back to me. I am assuming you have one on order?

Let me know if you would like this and if it would help!

Steven

plugwash
Forum Moderator
Forum Moderator
Posts: 3475
Joined: Wed Dec 28, 2011 11:45 pm

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 1:08 pm

If mpthompson doesn't want it and I haven't managed to get a Pi by other means by that time i'll take it.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 3:34 pm

veryevil said:


Hi, I just heard from farnell that my second Raspberry PI will be delivered on the 7/5/2012 and as I will then have two I would like to loan it out to you while you wait for yours.

I think the work you are doing here is spectacular and I thoroughly support it. I hope that having a PI sooner rather than later will help speed thinks up and help you out.

When you get the one you have ordered you can simply post it back to me. I am assuming you have one on order?

Let me know if you would like this and if it would help!

Steven


Steven, thank you for the very generous offer.  I do have two on order myself, but I don't know when I'll be getting them here in the states.  Actually, my work on this project isn't really hampered too much by not having a Pi since right now it is almost entirely about managing the package building process and other higher level issues.

Plugwash now has access to all the build systems and is really helping me out with this project. Since he is in the UK, perhaps it would be easiest/best to get the Pi to him and he can use it to verify the work we're both doing in the repository and create hard float images for others to use.  Either he or I having the Pi will help this project equally.

I see that Liz, and perhaps Eben, will be at the West Coast Makers Faire in a few weeks.  That's only a few miles from me.  If I don't have my Pi by then I'll see if I can meet them at the Faire and beg a Raspberry Pi off of them.  By then, hopefully a substantial part of the hard float repository will be created and this project will be kicking into high gear and get their attention.  I'm getting very curious as to whether having an RPi optimized version of Debian available is seen as useful to the Raspberry Pi Foundation members.  Things are still very premature, but that will hopefully be changing soon.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 4:12 pm

The Raspbian build servers have been up and running for just over a day and about 2000 new binary packages have been added to the repository.  Below is where we now stand with relation to the Debian armhf repository:

Debian Wheezy armhf package count: 34165
Raspbian Wheezy armhf package count: 19261

Don't get too excited by the 19261 number.  About 17000 of that number was the import of architecture 'all' binary packages into the repository that we didn't have to build.

I'll try to update these numbers on a regular basis as they are an easy to track metric.  If all goes according to plan, we should see the Raspbian Wheezy number slowly creep up to near parity with the Debian Wheezy number — hopefully by at least a few hundred a day, if not more.  We're still compiling the easy packages, but as we get into the more difficult/complex packages I expect we'll see days where the number only increases by a small amount.

The other thing to keep in mind is that Wheezy is a moving target.  We'll need to compile dozens of packages each day just to keep up with changes to new source packages being added to the Debian repository on a daily basis.  For now, we're not updating to keep things a little simpler while we get started.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Fri Apr 27, 2012 4:25 pm

wcchandler said:


I built a chroot of /wheezy to mess around with.  I know you have a nice build environment/setup, but I would like to help out if possible.  Do you have a list of packages that have yet to be compiled?  Might be fairly trivial to set it up on raspbian.  I completely understand if you want to do it all by yourself and just need someone to guinea-pig it out.  Not to mention your setup might not be conducive to outside help.

Either way let me know!


If you can email me at mpthompson(at)gmail.com I can discuss this offline with you.  Right now I'm a bit overwhelmed just getting things started, but I'm accumulating a list of packages that haven't built for various reasons.  It would be help if you could look over that list and start going through it to understand why certain packages don't build.  Some simply won't be supported by Debian armhf which I'm not too concerned with at this time, but others probably do reflect problems specific to our compilation.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Debian Hard Float (armhf) for RPi

Sat Apr 28, 2012 6:40 am

Counts as of Friday evening:

Debian Wheezy armhf package count: 34165
Raspbian Wheezy armhf package count: 20069

That's about 800 more packages built and added to the repository over the last 14 hours.  We are starting to reach some of the more minor X11 libraries so that tells me our building is now reaching deeper into the repository.  Unfortunately, we are seeing a substantial number of build failures with packages that attempt to link against certain libraries.  However, plugwash tracked those down to a misconfiguration of a linker setting in the gcc-4.6 compiler which should be fixed over the weekend.  That may fix a fair number of the 200 or so packages with build failures we've encountered over the last few days.

My hope is that by next weekend we'll be at 26000 to 28000 packages which would be roughly 2/3s of the architecture dependent packages that need to be built.  If that holds, by the middle of May we should be in a good position to start attempting full install of a Debian hard float distribution for the Raspberry Pi.

Surely there are some potential gottcha's still out there that could derail things for a while, but I'm feeling pretty good about the progress this week.  I'll sleep soundly tonight.

plugwash
Forum Moderator
Forum Moderator
Posts: 3475
Joined: Wed Dec 28, 2011 11:45 pm

Re: Debian Hard Float (armhf) for RPi

Sat Apr 28, 2012 11:18 am

IMO it's not so much about raw numbers (though they are certainly good to have) as it is about getting the RIGHT packages built, in particular at the moment i'm still having to resort to using the "dirty" chroot far more often than i'd like.

The biggest issue i'm running into is documentation generation tools (i'm currently trying tod deal with ghostscript), they tend to be build-depended on by a lot of packages and they tend to have a lot of dependencies themselves.

Return to “Raspbian”