Heater
Posts: 14683
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 8:51 am

Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?
Writing a decent compiler for any language is another huge task. Could take you years to learn how to do it and then implement it. Modern compilers are written and maintained by huge teams. See GCC, Clang/LLVM. Unless you happen to be some kind of genius like Fabrice Bellard https://bellard.org/tcc/

In short, if you want to write an OS, itself a huge job, use an existing language and compiler. Be it C, Rust, Pascal, whatever.
Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
Are there any resources to check for programming in Assembly? I would like to build the OS ground up. This is a very hard task, but I would like to program it AFTER building a prototype in another programming language.
Typically one does not write a "prototype" operating system in a high level language like C and then rewrite it in assembler. Operating systems have not been written in assembler since the 1960's. It would be a huge lot of work for useful reason.

You will almost certainly need to write some small parts of an OS in assembler. See the "Bare Metal" section of this forum for all kind of discussion on writing ARM assembler. There are ARM assembler tutorials all over the net.
Memory in C++ is a leaky abstraction .

jahboater
Posts: 5171
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 10:12 am

Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?
The direct answer to your question might be to use yacc (Yet Another Compiler Compiler) or Bison (the new GNU version).
A google search for "yacc grammar for c" will get you started.
For example:

https://www.lysator.liu.se/c/ANSI-C-grammar-y.html
Here is a C99 grammar:
https://gist.github.com/codebrainz/2933703

However I agree with Heater's comments, its not really sensibly practical - especially to write both a compiler and an OS.
GCC (the C compiler on Raspbian) has evolved over decades, by countless engineers, and been the subject of many university research projects.
The current register allocator (just part of the compiler) took 20 years to write.

The first popular, portable operating system was UNIX, written by Ken Thompson and Denis Ritchie (extremely clever and experienced engineers).
UNIX was first written in assembler, and then rewritten in C for portability and scalability. In fact they created C for the purpose.

More recently, Linux used a high level language from the start because by then C compilers were easily available.
Last edited by jahboater on Thu Feb 13, 2020 10:38 am, edited 3 times in total.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

jahboater
Posts: 5171
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 10:30 am

Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?
Note that Raspbian comes pre-installed with an excellent C compiler which can produce very fast executables and supports all the latest standards. Importantly it supports free standing programs that do not require a Linux host - which is what you need for writing an OS.

The assembler on the Pi is called "as". The linker is called "ld".
For fun, and to get you started with assembler, you could implement a tiny program such as hello world:

nano hello.s

Code: Select all

@
@  Print Hello world
@
	.text
	.global _start

_start:
	mov  r7,#4        @ write system call
	mov  r0,#1        @ file (stdout)
	adr  r1,message
	mov  r2,#14       @ message length
	svc  #0

	mov  r7,#1        @ exit system call
	mov  r0,#0        @ return code
	svc  #0

message:
	.ascii "Hello, world!\n"
then:

Code: Select all

$ as hello.s -o hello.o
$ ld hello.o -o hello
$ ./hello
Hello, world!
$ 
You may also use the compiler:

gcc hello.s -o hello

which has some benefits but produces a larger program.
You can use the C library and the C preprocessor (use hello.S)
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

Nirbodha
Posts: 6
Joined: Fri Feb 07, 2020 4:36 am

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 7:15 pm

Thank you for your reply!
Heater wrote:
Thu Feb 13, 2020 8:51 am
Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?
Typically one does not write a "prototype" operating system in a high level language like C and then rewrite it in assembler. Operating systems have not been written in assembler since the 1960's. It would be a huge lot of work for useful reason.

You will almost certainly need to write some small parts of an OS in assembler. See the "Bare Metal" section of this forum for all kind of discussion on writing ARM assembler. There are ARM assembler tutorials all over the net.
I did not know that. I did not really thoroughly research through the topic, so this is good to know.


I also saw your other post you posted later. Aren't the Bison and Yacc programs only for parsing?

-Nirbodha

P.S. as of my reading, I believe you are reaching the 5 grand post mark. Congrats.

jahboater
Posts: 5171
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 7:57 pm

Nirbodha wrote:
Thu Feb 13, 2020 7:15 pm
Aren't the Bison and Yacc programs only for parsing?
Yes. You need to add stuff to generate the assembler for each matched clause.
You need a lexical analyzer at the start (that "lex", or now the GNU "flex" can do for you).

Seriously, just use GCC!
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25367
Joined: Sat Jul 30, 2011 7:41 pm

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 7:58 pm

Nirbodha wrote:
Thu Feb 13, 2020 7:15 pm
Thank you for your reply!
Heater wrote:
Thu Feb 13, 2020 8:51 am
Nirbodha wrote:
Thu Feb 13, 2020 5:10 am
I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?
Typically one does not write a "prototype" operating system in a high level language like C and then rewrite it in assembler. Operating systems have not been written in assembler since the 1960's. It would be a huge lot of work for useful reason.

You will almost certainly need to write some small parts of an OS in assembler. See the "Bare Metal" section of this forum for all kind of discussion on writing ARM assembler. There are ARM assembler tutorials all over the net.
I did not know that. I did not really thoroughly research through the topic, so this is good to know.


I also saw your other post you posted later. Aren't the Bison and Yacc programs only for parsing?

-Nirbodha

P.S. as of my reading, I believe you are reaching the 5 grand post mark. Congrats.
Writing an OS is a HUGE task, requiring multiple people multiple years for anything approaching a usable system. After all, linux has over 10k contributors, and has taken 20 years.

You can write small real time OS's in a somewhat more sane timescale, but it's still measured in man years for anything that works well enough to use in anger.

If that hasn't put you off (it should do), then C would be the best option I reckon. Use the standard gcc compiler on the Pi to write bare metal code to get some practice in. Might get away with Rust, not sure if anyone has done bare metal stuff with it yet.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

jahboater
Posts: 5171
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 8:51 pm

Nirbodha,

If you want to write some system software, why not choose something smaller and more realistic?

Perhaps a tool like a text editor.
This could be written by one person in a few days (if you are good at C) and not 20 years / 10,000 programmers like Linux!

A text editor is an extremely useful tool that you will use many times a day, and writing your own means you can easily choose your own preferred screen layout and keyboard bindings. Very satisfying!
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

hippy
Posts: 6836
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 10:37 pm

I always enjoy these "cant be done; it takes many man years and many people" when it was all done in the past and didn't. That's driven by a false presumption that whatever emerges must be as good as what has emerged after all those man years when there's no evidence that's what a proposer is actually suggesting doing.

Perhaps the proposal is over-ambitious, ridiculously so, but has anyone asked exactly what the ambition is, what the scope of an OS is in the proposer's eyes ?

https://www.sigbus.info/how-i-wrote-a-s ... in-40-days

http://ultibo.org

Heater
Posts: 14683
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 14, 2020 7:09 am

I was about to write something similar.

In the modern perspective we might treat "I want to write as OS" as crazily over ambitious, especially for a beginner. Which it is even we interpret Operating System as being something like Linux or Windows.

But what if we scale down our definition of "operating system" to some basics:

1) Boot the bare machine into some running code: A boot loader.
2) Provide interfaces to some peripherals: Screen, keyboard, serial ports, storage.
3) Provide some user interface for loading code and running code, displaying output and interacting with it.
4) Perhaps debugging with memory/register dumps, run, stop, single step, break points.
5) Perhaps even multi-tasking.

At that scale of OS I claim to have written four operating systems:

Two in assembler for the 8 bit Motorola 6800 and 6809. Including all the above. Program storage was via a home made cassette tape interface.
One in PL/M 86. Including a co-operative multi-tasking scheduler. That even had wireless packet communications (No not WiFi)
One in assembler/C for the Intel 386. Including protected mode task switching with a simple scheduler. The hardest part there being the assembler required to get the thing booted and transition from 16 bit real mode to 32 bit protected mode.

An example of such an OS, for Intel x86 in Rust is here: https://os.phil-opp.com/
Or how about one for the Raspberry Pi, again in Rust, https://github.com/rust-embedded/rust-r ... -tutorials
Memory in C++ is a leaky abstraction .

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25367
Joined: Sat Jul 30, 2011 7:41 pm

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 14, 2020 8:20 am

Heater wrote:
Fri Feb 14, 2020 7:09 am
I was about to write something similar.

In the modern perspective we might treat "I want to write as OS" as crazily over ambitious, especially for a beginner. Which it is even we interpret Operating System as being something like Linux or Windows.

But what if we scale down our definition of "operating system" to some basics:

1) Boot the bare machine into some running code: A boot loader.
2) Provide interfaces to some peripherals: Screen, keyboard, serial ports, storage.
3) Provide some user interface for loading code and running code, displaying output and interacting with it.
4) Perhaps debugging with memory/register dumps, run, stop, single step, break points.
5) Perhaps even multi-tasking.

At that scale of OS I claim to have written four operating systems:

Two in assembler for the 8 bit Motorola 6800 and 6809. Including all the above. Program storage was via a home made cassette tape interface.
One in PL/M 86. Including a co-operative multi-tasking scheduler. That even had wireless packet communications (No not WiFi)
One in assembler/C for the Intel 386. Including protected mode task switching with a simple scheduler. The hardest part there being the assembler required to get the thing booted and transition from 16 bit real mode to 32 bit protected mode.

An example of such an OS, for Intel x86 in Rust is here: https://os.phil-opp.com/
Or how about one for the Raspberry Pi, again in Rust, https://github.com/rust-embedded/rust-r ... -tutorials
That list is still man years of work. Usb stack, plus various drivers it a huge amount of work by itself. Esp. On the pi4 since you need to write the pcie driver just to talk to the usb chip....
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Heater
Posts: 14683
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 14, 2020 6:25 pm

jamesh wrote:
Fri Feb 14, 2020 8:20 am
That list is still man years of work. Usb stack, plus various drivers it a huge amount of work by itself. Esp. On the pi4 since you need to write the pcie driver just to talk to the usb chip....
That is all very true.

But I did not say anything about USB or PCIe in my description of a minimal operating system. A UART will do, if if one has to bit-bang it.

After all, there was no USB or PCI stack in Linux when it was a twinkle in Torvalds' eye.
Memory in C++ is a leaky abstraction .

Nirbodha
Posts: 6
Joined: Fri Feb 07, 2020 4:36 am

Re: Creating an Operating System on the Raspberry Pi

Sat Feb 15, 2020 1:55 am

jahboater wrote:
Thu Feb 13, 2020 8:51 pm
Nirbodha,

If you want to write some system software, why not choose something smaller and more realistic?

Perhaps a tool like a text editor.
This could be written by one person in a few days (if you are good at C) and not 20 years / 10,000 programmers like Linux!

A text editor is an extremely useful tool that you will use many times a day, and writing your own means you can easily choose your own preferred screen layout and keyboard bindings. Very satisfying!

I believe this would be nice as a starter project. I will look into it. Thank you.

-Nirbodha

Return to “Beginners”