jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Building a network stack

Wed Sep 26, 2018 10:49 am

This is something I may be looking at in a month or 2 and was wondering if anyone else has done so and what pain points they had?

Assume I am working with a simple OS which handles memory, scheduling, etc.

Do I need a working USB stack before I can do anything with the ethernet controller?
If so, can I implement a minimal subset of USB for the sole purpose of driving the ethernet controller?
Once a USB stack is in place (assuming one is needed) how much work is a driver for the onboard ethernet controller?
Is it even feasible to write a driver for the ethernet controller from scratch? Or does everything just lift code from elsewhere?

I am just looking for a really rough idea of the work involved, and any major sticking points - so I can build a picture
of what, when and how!

Thanks

LdB
Posts: 1319
Joined: Wed Dec 07, 2016 2:29 pm

Re: Building a network stack

Wed Sep 26, 2018 12:24 pm

To drive only the ethernet you don't need a full USB stack all you need is a minimal implementation of a single channel on the USB.
You don't even have to enumerate the entire USB bus you could just enumerate the ethernet device and the Pi physical hub(s). Once enumerated you have given it an address which means you can fetch the endpoint via the end point descriptor read.

That endpoint on the ethernet chip is all you need to send and receive packets and poke it's registers. The rest of the USB can run dead at that point so long as the Pi physical hub endpoint can talk to the ethernet endpoint.

Normally we place a fake software roothub in at the bottom of the USB stack to handle messages like attaching and detaching devices even that is not required if all you want is the ethernet because the ethernet can not be physically attached/detached. In fact there are many many things that a full USB stack has that are irrelevant to the ethernet, obvious other things like handling slow devices which given the ethernet is on a fast endpoint to a fast physical hub is not required.

I just finished doing the LAN7800 driver on the Pi3B+ and I don't think there was much more than a dozen registers you actually need to bash on it to setup and the LAN7912 is actually simpler.

Receiving packets are actually trivial you will either poll or receive an interrupt. You will read it's length then transfer that amount of bytes out to some buffer and process. The first few bytes are the header which tells you whats in the packet. The write is a bit more work you have to put a header and some commands at the front of the data you wish to send and then send it out on the USB to the endpoint.

So it can be very simple if you like as a start and you can expand the complexity as you go.

Alternatively start with Xinu or XV6 which are already minimalistic O/S's that have Pi implementations and have the USB and ethernet running. They are fairly easy to understand assuming you have already built your own O/S. I messed around with both and from memory they were less than 1000 lines of code when you peeled them right back to minimals.

jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Re: Building a network stack

Wed Sep 26, 2018 12:58 pm

Thanks, that is exactly what I was looking for.

I am working with a Pi 3B+ - I am guessing this is the doc I need for the ethernet controller: http://ww1.microchip.com/downloads/en/D ... 01992F.pdf

I was looking for the documentation for the USB controller and ended up at Synopsys' website (from a link in the raspi peripheral doc). They listed a programmers guide (which looked perfect) however it is locked behind a signup which itself requires some sort of license code..

Do you know where I can find all the relevant documentation I need for the USB controller?

Thanks!

ghans
Posts: 7878
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Building a network stack

Wed Sep 26, 2018 2:34 pm

The USB controller on the Pi has no openly available documentation. Everybody has had to reverse-engineer it using existing drivers (the first ones had to use the Linux one ... several thousand lines of it).

Synopsys considers Broadcom and only Broadcom as its customer, not Raspberry Pi, much less natural persons who bought Raspberries themselves.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

LdB
Posts: 1319
Joined: Wed Dec 07, 2016 2:29 pm

Re: Building a network stack

Wed Sep 26, 2018 4:02 pm

jamesmintram wrote:
Wed Sep 26, 2018 12:58 pm
Do you know where I can find all the relevant documentation I need for the USB controller?
As stated most are just conversions of linux drivers, there really isn't any real documentation

I did a redux of chadderz version and I thought that was slim
https://github.com/LdB-ECM/Raspberry-Pi ... m32_64_USB

However I recently ran across the Plan 9 driver and that is even more reduxed
https://github.com/brho/plan9/blob/mast ... m/usbdwc.c

It is multi channel but it basically does just enough to do what you want

Code: Select all

 * This is work in progress:
 * - no isochronous pipes
 * - no bandwidth budgeting
 * - frame scheduling is crude
 * - error handling is overly optimistic
 * It should be just about adequate for a Plan 9 terminal with
 * keyboard, mouse, ethernet adapter, and an external flash drive. 
It's less than a thousand lines of code half of which are not relevant to you and you can comment out because they are system resource locks and debugging. As I understand it you are running one app so you won't need resource locks unless you really go mad. I can't imagine you are going to get up to a point you end up using all 8 channels so most of the locks won't ever come into play.

It doesn't automatically do the enumeration of the bus but once you understand the steps it's easy to make that in a small code function. That is about the only thing missing.

I am doing a conversion of it because I like the format but slightly bloated (more error reporting) and with C atomic semaphores and call backs so I can use it on anything and it is setup for multitasking.

jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Re: Building a network stack

Wed Sep 26, 2018 4:27 pm

Shame about the documentation not being available :(

Those resources are really great though, small enough to easily get the big picture.

Thanks

ronyoung1
Posts: 6
Joined: Fri Oct 11, 2013 6:37 am

Re: Building a network stack

Wed Sep 26, 2018 6:43 pm

In addition to the USB stuff discussed, you may also want to look at lwip (lightweight IP) stack.
It is located at: https://savannah.nongnu.org/projects/lwip/

-ron

jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Re: Building a network stack

Wed Sep 26, 2018 7:06 pm

Thanks ronyoung1 - that looks really great!

I am looking forward to working on this :)

rst
Posts: 414
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: Building a network stack

Wed Sep 26, 2018 7:32 pm

jamesmintram wrote:
Wed Sep 26, 2018 10:49 am
This is something I may be looking at in a month or 2 and was wondering if anyone else has done so and what pain points they had?
My experience is, doing this from scratch is not that easy and it takes time, but you can learn a lot from it about USB 2.0 and TCP/IP.
Do I need a working USB stack before I can do anything with the ethernet controller?
If so, can I implement a minimal subset of USB for the sole purpose of driving the ethernet controller?
Once a USB stack is in place (assuming one is needed) how much work is a driver for the onboard ethernet controller?
Is it even feasible to write a driver for the ethernet controller from scratch? Or does everything just lift code from elsewhere?
Yes, you need an USB host controller interface (HCI) driver with reduced function. For the Ethernet controller you need USB bulk transfers and USB control transfers for device initialisation. You also have to initialise the on-board USB hub, before you can access the Ethernet device. If you do not want to access an USB keyboard, you do not need USB split transfers, which makes it easier. The Ethernet controller driver itself is relatively simple and small compared to the other USB stuff. I would suggest to have a look at the sticky "Bare metal resources" thread on top of this forum. There are several different bare metal USB drivers available today, which have all advantages and disadvantages. You have to use them to get information, because a specification about the host controller hardware is not available for the public.

The network stack itself is a thing of its own. Once you have found a model, how to process the different network layers in your system, it is a lot of coding and testing to get it running. Especially the TCP protocol handler is difficult to become stable. So it's probably not a bad idea to reuse an existing network stack, unless you have special requirements or you want to learn.

jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Re: Building a network stack

Wed Sep 26, 2018 8:02 pm

Hi rst, thanks for that, there is some really good information there.

My experience is, doing this from scratch is not that easy and it takes time, but you can learn a lot from it about USB 2.0 and TCP/IP.

I actually meant I will be ready to start working on the USB/Network stack in a month or two. I am still working on the memory system and process management, but I wanted to start reading around the subject before I am ready to start.

I suspect this will take me 6 - 9 months to actually get something not completely broken. It's purely as an educational thing.

dwelch67
Posts: 961
Joined: Sat May 26, 2012 5:32 pm

Re: Building a network stack

Wed Sep 26, 2018 10:17 pm

lwip is a popular solution. As mentioned above it depends on whether you have a reason to write your own. And at what level. ARP and ping will be your first test to get your feet wet. With practice you can bang out a (special purpose) UDP stack (with cheats) in an afternoon. And when I say that I have done some that are slave only so some master/server sends packets to me, I cheat and dont arp I just flip the source/destinations in the header. and send a response, I even go so far as to fix the packet size. Not in any way a full stack but you can have a normal full stack host computer talk to you for some purpose. Real udp you need to arp you need to timeout your arp table entries. etc, etc. TCP is harder even receiving one connection, one packet sending one back and cleaning up so that the next connection works will take some time and thats assuming no fragmentation. There are some lighter weight protocols in between that you could look for (SCTCP for example), YMMV. Depends on how much of this is an educational task, the whole thing or just the OS parts and not necessarily the drivers. I recommend though once you get the USB up you should try to do RARP and ping yourself, not difficult, educational, and there is some level of pride in being able to ping to your code from some other place. Yyes there is a command at least in linux to re do or clear the arp table so that you can get multiple arps and not have to keep changing your ip address or mac address. After rarp and ping then look at udp and tcp and see if you really want to get into that vs integrating lwip.

As far as IP goes this is very typical. DRAM, USB, ethernet, video. IP like that which is non-trival is usually something you purchase, from companies like cadence since you are already giving them a million to a few million dollars a year to use their tools to make the chip in the first place whats another few ten thousand? And this IP generally has an NDA, so if you are a company that provides documentation to customers you still cant provide documentation for that IP. I find it difficult to provide open source software for NDA protected items as to me that is providing documentation about the IP. The good-ish side of this is that "everyone" buys the same IP from the same vendor(s). So as mentioned above you will find multiple chips from one or more vendors that use the same or similar open source drivers or their drivers and some other driver basically do the same thing even if it isnt the same source code. You dont see this at all or as much on a microcontroller one because they dont normally need these complicated interfaces, the usb is usually buried in the logic you dont have direct control, same with ethernet, and you dont normally have dram, dont have high speed phys or serdes to deal with. And that market historically you write your own baremetal code. Markets where you have operating system capable chips (not that an mcu cant run an RTOS) like full blown linux, windows, etc. The chip vendor makes a board support package has written the drivers or taken code from the IP vendor and used it directly or indirectly, whatever and the rest of us just write printf() programs on the operating system for that platform, applications. broadcom, allwinner and some others generally dont want any of their documentation out ,they have NDAs with customers. note buying a raspberry pi we are not a customer. the raspberry pi folks (Which are I think actually broadcom) are a customer and they can see the real documentation. Some allwinner customers have "accidentally" left marked documents laying around, and in recent years seems like some have had permission to re-release. Folks like ti I am happy to see have docs you can just download. But I assume you still cannot see the ddr nor usb nor phy/serdes register specs, maybe the ethernet, maybe not. I have not looked into that portion of those docs, was only interested in booting and blinking some leds and spitting characters out the uart.

Each new process can dictate new IP for a chip vendor, as well as a new foundry can dictate new ip. The dram and usb and ethernet controllers are or can be portable digital logic, but they tie into phys that are part analog and have to be designed to the process. LIkewise time passes. You might not necessarily be able to find a DDR2 controller/phy that works on a 10nm process at xyz foundry. Maybe not even DDR3 but certainly DDR4. Same goes for usb1.0, pcie gen1 only, etc, etc. Why would you build a new chip on a current process that only supports old protocols or speeds? How intimate the controller and the phy are and/or how interested you are in creating a standards compliant usb controller vs just buying one is how we get in this situation. Its an endless cycle. I am happy to see that baremetal is a thing on the pi, but it really isnt a thing on other similar platforms, dont know why it turned out this way.

Why do companies hide this info? From low level ip blocks to chips to boards? Various reasons in theory protecting something, from cloning or other "secrets". But if I have to call some regional salesperson and give up my name, address, name of first born, etc just to get a document I should have been able to download and as a reward you call me every week or show up at the office to harass me or try to get names of other employees to harass. I have personally blacklisted such companies, wont buy from them, I am certainly not interested in your product. It cant be that special. Arguably if broadcom had not put out the arm document they did, I probably wouldnt be here, would never have had a reason to buy a raspberry pi. Only in the last year or two have I bothered to look at anything allwinner for that reason. But the dram/usb/ethernet/etc IP though I expect that to be closed document even for actual chip customers who pay the big bucks to Broadcom and wouldnt personally blacklist them for that being closed source. At least now that I have been in the chip business for a while. Before knowing all of this I probably would have. Fortunately I dont buy stuff other than for homebrew hobbies at this point otherwise where I work we probably wouldnt have any products I would have blacklisted them all for one reason or another.

Back to ethernet.

Even though the RFCs say this one is superseeded by that one, find the older/oldest ones first. And/or google

tcp/ip illustrated volume 1

This should contain those basics and if I remember right tells you the RFC numbers.

Hint: The checksum for UDP and maybe TCP I think they call it a ones complement. You have to roll the carry out back in to get the right answer.
Hint: For UDP it says you can leave the checksum zero and that is valid (dont have to compute it until you are ready to swap/fix the mac address swap the ip addresses, change the payload, zero the checksum, send back out).

TCP is PITA...

jamesmintram
Posts: 31
Joined: Mon Jan 15, 2018 12:14 am

Re: Building a network stack

Thu Sep 27, 2018 7:14 am

Hey dwelch67,

Thanks for all that info!

ARP and ping will be your first test to get your feet wet

Sounds like ARP and Ping are similar to rendering your first trianglewhen building a graphics engine!

tcp/ip illustrated volume 1 This is on my list to purcahse, also I already have Tanenbaum. Those
coupled with some source code should get there.

Regards,
James

rst
Posts: 414
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: Building a network stack

Thu Sep 27, 2018 11:45 am

jamesmintram wrote:
Wed Sep 26, 2018 8:02 pm
I actually meant I will be ready to start working on the USB/Network stack in a month or two. I am still working on the memory system and process management, but I wanted to start reading around the subject before I am ready to start.

I suspect this will take me 6 - 9 months to actually get something not completely broken. It's purely as an educational thing.
Good plan. It depends on your knowledge, but in this space of time you should be able to develop something, which is working well.

Return to “Bare metal, Assembly language”