travistrue2008
Posts: 20
Joined: Thu Oct 05, 2017 6:09 pm
Location: Las Vegas
Contact: Website Facebook

How Much of the Raspbian Stack is Open Source?

Sat Apr 14, 2018 7:30 pm

I'm learning more about running bare-metal code, assembly, and the Linux stack (kernel, DTB, kernel modules, GNU environment, etc). I thought that Raspbian would be a good start since I want to learn how to write assembly in ARM. I think it would be a fun project to write a micro-kernel in Linux using Rust that compiles on the Pi Zero W, and port a few of GNU's coreutils programs over to Rust. I know some of libraries for the Pi build of Raspbian is closed-source due to Broadcom's required NDA with the Pi Foundation, and I wouldn't doubt it if there is proprietary software written by the Pi Foundation to support the Pi boards that can't be open-sourced due to these NDAs. Due to this, there are certain libraries (such as the VideoCore libraries) that I wouldn't be able to rewrite in Rust. I'd have to install the C libraries that are currently provided, and write wrapper libraries in Rust if I wanted to support that.

What I'd like to do is break down the Stretch Lite build of Raspbian into the major contributors, and group them by open-source, or closed-source. Here's what I've come up with so far:

Open-Source
Kernel/Drivers (the Pi Foundation's Debian fork)
Device Tree Blobs (part of the Pi's firmware)
GNU Coreutils (Debian)

Closed-Source
Bootloader (is this provided by Broadcom, or Pi Foundation?)
VideoCore firmware (Broadcom)

Are there any major components that am I missing?
Last edited by travistrue2008 on Sun Apr 15, 2018 6:15 pm, edited 1 time in total.
Sic parvis magna.

User avatar
Paeryn
Posts: 2230
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: How Much of the Raspbian Stack is Open Source?

Sat Apr 14, 2018 8:43 pm

Device Tree stuff isn't closed source, the source files are in the kernel source (that might not be the correct kernel version, can't remember off the top of my head which we are on).

The bootloader I would say is part Broadcom part RPF (most likely RPF alterations to a base provided by Broadcom), this runs on the VC4 itself and pretty much other than reverse-engineered stuff we've no way to write for that, Broadcom have never released details on the VC4's processors.

The only bits of the VC4 that we know about that I can think of is the basic peripherals and the 3D core (though I think there is some information about the display controller in Eric Anholt's work on the experimental OpenGL driver).
She who travels light — forgot something.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5373
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: How Much of the Raspbian Stack is Open Source?

Sat Apr 14, 2018 9:50 pm

Also, although the firmware that runs on videocore is closed source, the userland libraries to talk to that firmware are open. In many cases, they're fairly simple functions that tell the firmware to do the heavy lifting, which some people don't like. Anholt has been doing a lot of work replacing some of the firmware components with open source alternatives than run on the ARM.

Long story short is that there are no closed components that would prevent your bare metal kernel from doing everything raspbian can do. Ultibo provide a good example of what's possible with bare metal on the pi.

ghans
Posts: 7819
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: How Much of the Raspbian Stack is Open Source?

Sun Apr 15, 2018 10:00 am

The QPU units inside the GPU have freely available documentation.

https://docs.broadcom.com/docs/12358545

I suppose Anholts work would be unthinkable without publication of that document, which was unprecendented for Broadcom and the mobile GPU space in general.

ghabs
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

travistrue2008
Posts: 20
Joined: Thu Oct 05, 2017 6:09 pm
Location: Las Vegas
Contact: Website Facebook

Re: How Much of the Raspbian Stack is Open Source?

Sun Apr 15, 2018 7:08 pm

Thanks everyone!

I've updated my answer to reflect that the DTBs are open, and specified that it's the GPU's firmware that isn't open-source. That's good to know that everything in Raspbian is open-source.

That makes sense that the user-space libraries are merely wrappers to talk to the GPU's firmware. How do those libraries go about sending commands to the GPU's firmware? Is it through the mailbox API?

Does this mean that the function implementations for the OpenGL ES 2.0 calls in libbrcmGLESv2.so are function wrappers to construct the real query that gets sent to the GPU's firmware via the Mailbox API? That's my theory since the library contains hundreds of function implementations, yet it's only 103KB. I haven't found the source code for libbrcmGLES2v.so, so my guess is that it's closed-source, and possibly not even a user-space library.

I've looked into a little of Eric Anholt's work in the past. He's a principle engineer at Broadcom, right?
Sic parvis magna.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20946
Joined: Sat Jul 30, 2011 7:41 pm

Re: How Much of the Raspbian Stack is Open Source?

Sun Apr 15, 2018 7:14 pm

travistrue2008 wrote:
Sun Apr 15, 2018 7:08 pm
I've looked into a little of Eric Anholt's work in the past. He's a principle engineer at Broadcom, right?
I believe so.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

ghans
Posts: 7819
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: How Much of the Raspbian Stack is Open Source?

Sun Apr 15, 2018 7:25 pm

The source for libbcrmGLESv2.so must be here:
https://github.com/raspberrypi/userland ... ronos/glxx
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

User avatar
Ultibo
Posts: 139
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: How Much of the Raspbian Stack is Open Source?

Mon Apr 16, 2018 11:22 am

travistrue2008 wrote:
Sun Apr 15, 2018 7:08 pm
How do those libraries go about sending commands to the GPU's firmware? Is it through the mailbox API?

Does this mean that the function implementations for the OpenGL ES 2.0 calls in libbrcmGLESv2.so are function wrappers to construct the real query that gets sent to the GPU's firmware via the Mailbox API? That's my theory since the library contains hundreds of function implementations, yet it's only 103KB. I haven't found the source code for libbrcmGLES2v.so, so my guess is that it's closed-source, and possibly not even a user-space library.
The communication with the GPU for OpenGL, OpenVG, OpenMAX, MMAL, CEC and a lot of other stuff (eg vcgencmd) is done using the VCHIQ driver. This is a shared memory protocol between the ARM and the GPU which passes messages and blocks of data back and forth between the two, the mailbox is not involved except during initialization.

The user side libraries communicate through an ioctl to make requests to the vchiq, all of the magic is done in the kernel driver which is here https://github.com/raspberrypi/linux/tr ... 4_services

The .so files (like libbrcmGLES2v.so) are all compiled from the userland repo and are definitely open source.

You can implement it all without Linux, it takes a lot of work but we proved it is possible.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

Return to “General programming discussion”