Nirbodha
Posts: 15
Joined: Fri Feb 07, 2020 4:36 am

Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 4:39 am

Hello. I received a Raspberry Pi 4 B+, and I was wondering if I could create my own operating system.

Are there any good leads for me to follow to create an OS?
If you haven't known already, I'm Nirbodha.

Nirbodha means something along the lines of being dull in Bengali. As such, expect my responses to sound like a blockhead made them. Thanks.

tpyo kingg
Posts: 809
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 7:13 am

If you mean creating from scratch then you might look at Minix, Haiku-OS, or even FreeDOS for inspiration.

If you mean rolling your own distro, or even just a respin, then you can use Raspbian as your base. The guides and tools that apply to Debian will be relevant for Raspbian. Linux Respin is a fork of the late remastersys. However, neither are in the repository, it too will have to be built manually.

What are your goals with building a custom distro or respin?


Heater
Posts: 15507
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 8:28 am

Nirbodha ,
Are there any good leads for me to follow to create an OS?
Yes:

1) Learn to program in Rust: https://www.rust-lang.org/

2) Write operating system for the Pi: https://github.com/rust-embedded/rust-r ... -tutorials

You might want to checkout some books on OS design:
https://www.amazon.com/Design-UNIX-Oper ... 0132017997
https://www.amazon.com/Modern-Operating ... 013359162X
Memory in C++ is a leaky abstraction .

User avatar
DougieLawson
Posts: 38549
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 8:44 am

Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

I'll do your homework for you for a suitable fee.

Any DMs sent on Twitter will be answered next month.
All non-medical doctors are on my foes list.

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 9:19 am

Learn C

All the successful, widely used, operating systems are written in C for good reason (e.g. Windows, Linux, MacOS, UNIX etc).
The C language was created specifically for writing operating systems (and compilers, editors, and other system tools).

Some knowledge of assembler may be useful too.

See the "bare metal" section in this forum.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

hippy
Posts: 7177
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 9:46 am

Or take a look at Ultibo and learn Pascal.

Heater
Posts: 15507
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 11:58 am

jahboater wrote:
Fri Feb 07, 2020 9:19 am
Learn C

All the successful, widely used, operating systems are written in C for good reason (e.g. Windows, Linux, MacOS, UNIX etc).
The C language was created specifically for writing operating systems (and compilers, editors, and other system tools).
...
That is true. C has a great track record as being an OS implementation language. Mostly because of the great example set by Unix as you say. As far as I can tell subsequently because it was pretty much the only viable choice.

However, that all started forty years ago! It's staggering to think that in all that time we never came up with anything better, especially in the face of all of C's obvious shortcomings and the enormous progress that has been made in every other area of technology.

Is it so this situation is not going to change, forever?
Memory in C++ is a leaky abstraction .

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 6:47 pm

Heater,
Heater wrote:
Fri Feb 07, 2020 11:58 am
Is it so this situation is not going to change, forever?
I hope not.

C definitely has evolved of course - but slowly and carefully.
Its a far safer, more portable, and more usable language than K&R was - all done without invalidating all the billions of lines of existing code.
(unlike Python's history for example).
After nine years I still have not used all the stuff in C11 (like you, I only recently learned of the new atomics and threads for example).

All the newer languages that come along are (one or more of):

Too large and complicated
Safer at the expense of performance.
Lack widespread availability.
Lack a strong ISO standard and strictly conforming compilers.
Lack capability (too many languages rely on C for their run time library etc).
Lack transparency (the code must do exactly what it says and no more, no GC)
Lack precise control over the data types in use.
Lack basic scalar types such as narrow or unsigned integers.
Require some special, non-standard, build environment.
Just lack performance.

The OS code must be fast, very fast.
For example software and hardware interrupts might occur thousands of times per second and the system must handle them quickly or everything slows down. You can imagine how heavily optimized open/read/write/close are.
And so on.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

W. H. Heydt
Posts: 12120
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 7:54 pm

Heater wrote:
Fri Feb 07, 2020 11:58 am
That is true. C has a great track record as being an OS implementation language. Mostly because of the great example set by Unix as you say. As far as I can tell subsequently because it was pretty much the only viable choice.

However, that all started forty years ago! It's staggering to think that in all that time we never came up with anything better, especially in the face of all of C's obvious shortcomings and the enormous progress that has been made in every other area of technology.
More like 50 years ago by now.
Is it so this situation is not going to change, forever?
Student exercise... C is derived from B. B is derived from BCPL. What is the next language called?

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 8:29 pm

W. H. Heydt wrote:
Fri Feb 07, 2020 7:54 pm
Student exercise... C is derived from B. B is derived from BCPL.
and BCPL was derived from CPL (Basic CPL).
W. H. Heydt wrote:
Fri Feb 07, 2020 7:54 pm
What is the next language called?
You mean the debate: should it be called D or P ?
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

W. H. Heydt
Posts: 12120
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Creating an Operating System on the Raspberry Pi

Fri Feb 07, 2020 8:37 pm

jahboater wrote:
Fri Feb 07, 2020 8:29 pm
W. H. Heydt wrote:
Fri Feb 07, 2020 7:54 pm
Student exercise... C is derived from B. B is derived from BCPL.
and BCPL was derived from CPL (Basic CPL).
W. H. Heydt wrote:
Fri Feb 07, 2020 7:54 pm
What is the next language called?
You mean the debate: should it be called D or P ?
The question is straight out of the Kernighan & Ritchie "white book". But, yes, that is how I would parse it.

Heater
Posts: 15507
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Mon Feb 10, 2020 3:10 pm

jahboater wrote:
Fri Feb 07, 2020 6:47 pm
C definitely has evolved of course - but slowly and carefully.
Its a far safer, more portable, and more usable language than K&R was - all done without invalidating all the billions of lines of existing code.
(unlike Python's history for example).
After nine years I still have not used all the stuff in C11 (like you, I only recently learned of the new atomics and threads for example).
In what way is C today any "safer" than it was back in K&R days?

In what way is C more portable than it was in K&R days? Clearly C is available for just about anything today but I don't see how any changes in the language definition have aided that.

Atomics and Threads are nice. An acknowledgement that the original C concept of the machine it runs on does not hold up today. We have many processors not just one, memory is not as simple as the C model assumed.

But ironically Atomics and Threads in C are just syntax over underlying mechanisms like pthreads. One is not going to be using C's atomics and threads when writing an operating system. After all, one has to create those mechanisms in the OS one is writing, in C !
Memory in C++ is a leaky abstraction .

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Mon Feb 10, 2020 6:22 pm

Heater wrote:
Mon Feb 10, 2020 3:10 pm
In what way is C today any "safer" than it was back in K&R days?
Trivial example: in the bad old days people used to do all kinds of questionable things like storing pointers in integers (probably because that was what you did in B and BCPL). That sort of thing is now unthinkable.
The "const" keyword really helps. New stuff in C11 like explicit control over alignment, etc.
To be fair to your point, modern advances in compilers provide superb diagnostics giving the "impression" of a safer language.
For example assigning an "int" to an "unsigned int" gets a nasty warning, I am sure it never did 40 years ago.
Heater wrote:
Mon Feb 10, 2020 3:10 pm
In what way is C more portable than it was in K&R days? Clearly C is available for just about anything today but I don't see how any changes in the language definition have aided that.
Here's one from C99: "int64_t" works identically on all platforms regardless of hardware support.
It is defined to be twos-complement and 64-bits wide.
Prior to that, a long had to be "at least as large as an int", with no guarantee of its characteristics.
Integer arithmetic is now truly portable.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

User avatar
jojopi
Posts: 3194
Joined: Tue Oct 11, 2011 8:38 pm

Re: Creating an Operating System on the Raspberry Pi

Mon Feb 10, 2020 7:19 pm

jahboater wrote:
Mon Feb 10, 2020 6:22 pm
Here's one from C99: "int64_t" works identically on all platforms regardless of hardware support.
ISO/IEC 9899:* §7.[18|19|20].1.1.3 These types are optional. However, if an implementation provides integer types with widths of 8, 16, 32, or 64 bits, no padding bits, and (for the signed types) that have a two's complement representation, it shall define the corresponding typedef names.

int64_t is not guaranteed to exist, and even if it does exist it may be slower than the native types. "long long" is guaranteed to exist, and to be at least 64 bits, and is probably the most efficient integer of that size.

intN_t are not intended for general usage; they are for code that requires very specific wrapping and overflow behaviour, or structure packing for interchange.

If you are sure you are on a byte-oriented machine, then why not just use the native types?

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Mon Feb 10, 2020 7:48 pm

jojopi wrote:
Mon Feb 10, 2020 7:19 pm
jahboater wrote:
Mon Feb 10, 2020 6:22 pm
Here's one from C99: "int64_t" works identically on all platforms regardless of hardware support.
ISO/IEC 9899:* §7.[18|19|20].1.1.3 These types are optional. However, if an implementation provides integer types with widths of 8, 16, 32, or 64 bits, no padding bits, and (for the signed types) that have a two's complement representation, it shall define the corresponding typedef names.
Yes. Good point. But all the popular compilers provide them and have done for many years.
The word "shall" above is interesting: since "long long" is guaranteed to exist, it implies that int64_t must also exist.

Put another way. If a compiler can provide 64-bit integer arithmetic that it must define int64_t. Since you say that all C compilers must provide long long which is 64-bits, then it follows that they must always define int64_t. So the standard is wrong, its not optional ?

Note that its also theoretically possible for long long to be larger than 64-bits ....
(but I suspect that's just as unlikely as stdint being missing).
jojopi wrote:
Mon Feb 10, 2020 7:19 pm
it may be slower than the native types. "long long" is guaranteed to exist, and to be at least 64 bits, and is probably the most efficient integer of that size.
It uses the native types anyway (at least GCC does judging by the printf formats it chooses), so I really dont see why it would be slower.
I have a 12,000 line program here that makes extensive use of int64_t (to avoid having to check for overflow).
On my Intel PC I changed int64_t to "long". The resulting executable is identical.
On my Pi4 (32-bit) I changed int64_t to "long long". Again, the resulting executable is identical.
jojopi wrote:
Mon Feb 10, 2020 7:19 pm
If you are sure you are on a byte-oriented machine, then why not just use the native types?
Because I want portability. For example: on nearly all platforms "int" is 32-bits but on a few it is only 16 bits. Quite easy to overflow.
So I would use int32_t to avoid the hassle of dealing with that and ensure my source code is the same everywhere.
I suppose you can do:

static_assert( sizeof(int) >= 4, "int size should be at least 32 bits" );

instead :)
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

User avatar
jojopi
Posts: 3194
Joined: Tue Oct 11, 2011 8:38 pm

Re: Creating an Operating System on the Raspberry Pi

Tue Feb 11, 2020 4:28 am

jahboater wrote:
Mon Feb 10, 2020 7:48 pm
The word "shall" above is interesting: since "long long" is guaranteed to exist, it implies that int64_t must also exist.
"long long" could be 72 bits, include padding, or be stored sign-magnitude. In these cases, int64_t will either not exist, or it will be slower than the native types.

Of course, I do not blame you if you only care about two's complement, octet, machines with sane compilers. But in that case: int8_t is just signed char, int16_t is short, and int64_t is long long. The only question is whether int32_t corresponds to native int or long.

Using int32_t unnecessarily on an 8 or 16 bit platform will be slower than int. Conversely, using int16_t may be slower than int on a 32 bit platform. (RISC processors often do not provide narrow registers or arithmetic, so the compiler has to throw in extra mask instructions.) The native types are the best types for the platform, and we should not be trying to avoid them.

Again, exact-width types are strictly for cases where the algorithm would be wrong if the type were not the exact width.

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Tue Feb 11, 2020 10:43 am

jojopi wrote:
Tue Feb 11, 2020 4:28 am
jahboater wrote:
Mon Feb 10, 2020 7:48 pm
The word "shall" above is interesting: since "long long" is guaranteed to exist, it implies that int64_t must also exist.
"long long" could be 72 bits, include padding, or be stored sign-magnitude. In these cases, int64_t will either not exist, or it will be slower than the native types.
For 72-bits your are presumable reffering to Honeywell mainframes of 40+ years ago (which I used to use). And yes, its still around - as a software emulation in a museum!
How many machines in production today have 72-bit registers? Nil I suspect. 128-bits I think may happen for some rare cases in the future (but I bet int64_t will still be defined).

There are no known computers now that do not use twos-complement. C and C++ have traditionally regarded signed integer overflow as "undefined behavior". However, because integers are always twos-complement, C++14 has changed and signed overflow is now "defined" (and therefore safe to exploit). C is planned to go the same way in C2x. You will only see sign-magnitude in Javascript which uses the floating-point registers to store integers!
jojopi wrote:
Tue Feb 11, 2020 4:28 am
Of course, I do not blame you if you only care about two's complement, octet, machines with sane compilers. But in that case: int8_t is just signed char, int16_t is short, and int64_t is long long. The only question is whether int32_t corresponds to native int or long.
Interesting. I see what you mean. One less include file, and the printf formats would be cleaner. "%lld" looks better than "%" PRId64
You might possibly have convinced me!
jojopi wrote:
Tue Feb 11, 2020 4:28 am
Using int32_t unnecessarily on an 8 or 16 bit platform will be slower than int.
Yes of course. The same goes for int64_t on a 32-bit platform.
I just accept it for these unusual cases.
However, (1) it means my code is simpler, cleaner, and safer because I don't need to worry about overflow. (2) my source code is the same for all target platforms (3) structures can be an optimal size. (4) I don't have any messy #ifdefs to special case code for rare word sizes.
jojopi wrote:
Tue Feb 11, 2020 4:28 am
Conversely, using int16_t may be slower than int on a 32 bit platform. (RISC processors often do not provide narrow registers or arithmetic, so the compiler has to throw in extra mask instructions.) The native types are the best types for the platform, and we should not be trying to avoid them.
Yes. Agreed. I use int16_t only for storage to save space.
But! To evaluate an expression C promotes anything smaller than an int, to an int, so arithmetic is automatically done in the native size.
Probably for just this reason. Actually the ARM on the Pi is pretty good with 16-bit arithmetic, you just lose the ability to detect overflow or carry.
It does ldrh and strh to load and save 16-bit values and does not do any masking. Sign extension may also be automatic.
Last edited by jahboater on Tue Feb 11, 2020 11:20 am, edited 1 time in total.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

Heater
Posts: 15507
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Tue Feb 11, 2020 5:39 pm

jahboater,

Ah, OK. "const" and the integer types are nice. Not much but good to have.

I seem to recall that the first C compiler I used, from MS, in 198'something would make code out of pretty much anything one typed into it. God knows what it might do at run time. I was quite shocked at this behavior having been weaned on BASIC and ALGOL. And taken aback that the group I had joined thought this was normal. I soon realized I was dealing with some weird kind of assembler and got with it OK after that. Slowly coming to believe this kind of sloppiness in a language system was normal too. Until there was Rust, to remind me I was not wrong to be shocked.

The discussion above about what is an integer and what integer type to use reinforces my view that C is still sloppy.

128 processors and 128 bit C might be coming soon. The RISC V instruction set has a 128 bit extension specified. Originally specified as a kind of joke, and because it fits the RISC V plan nicely. Turns out there are those in the super computer and security worlds that are considering 128 bit seriously.
Memory in C++ is a leaky abstraction .

jahboater
Posts: 5492
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: Creating an Operating System on the Raspberry Pi

Tue Feb 11, 2020 6:04 pm

Heater,
Heater wrote:
Tue Feb 11, 2020 5:39 pm
The discussion above about what is an integer and what integer type to use reinforces my view that C is still sloppy.
Rust has i64 u64 i16 u16 etc which are pretty much the same as the C99 integer types (except a bit terse).
The difference is that Rust is a new language and can jump straight in with all these "powers of two" sizes because that is what modern hardware has available, almost universally.
C has evolved over 48 years and had to deal with a vast range of hardware from tiny embedded micro controllers up to super computers. Many of these had weird word sizes, 9-bit bytes and so on. Late in the day, well 20 years ago, C introduced the exact width "power of two" sizes that Rust has now.
I doubt Rust would work well on a mainframe with 60 bit or 36 bit words etc - it has no suitable data types, but I believe C would still cope and be fast.

C is pragmatic rather than sloppy.
"We are in the beginning of a mass extinction, and all you can talk
about is money and fairy tales of eternal economic growth."
- Greta Thunberg

Heater
Posts: 15507
Joined: Tue Jul 17, 2012 3:02 pm

Re: Creating an Operating System on the Raspberry Pi

Tue Feb 11, 2020 8:14 pm

Quite so. It all makes perfect sense why C was defined as it was.

Pragmatically sloppy :)
Memory in C++ is a leaky abstraction .

mwrich4
Posts: 90
Joined: Wed Jul 25, 2012 1:24 pm
Location: Stuart, Florida

Re: Creating an Operating System on the Raspberry Pi

Wed Feb 12, 2020 3:54 am

Nirbodha wrote:
Fri Feb 07, 2020 4:39 am
Hello. I received a Raspberry Pi 4 B+, and I was wondering if I could create my own operating system.

Are there any good leads for me to follow to create an OS?
It is best to have a wealth of experience from having used a variety of operating systems before creating your own.

Studying FreeRTOS is the best practical way to learn about coding some of the mechanisms operating systems rely on. Taking a Comp Sci curriculum first is even better.

It is NOT an easy nor quick task. It takes a tremendous familiarity with the host CPU to make an effective OS. For learning, those concerns are minimal.

Nirbodha
Posts: 15
Joined: Fri Feb 07, 2020 4:36 am

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 5:10 am

Woah guys, thanks for the replies! Me being a newbie in these forums, I couldn't find this post until now.


I do want to create a completely original operating system from scratch. I did see a few comments talking about using C. I do have experience in programming in it. If I were to use it though, how would I implement a compiler for C to the Pi?

Are there any resources to check for programming in Assembly? I would like to build the OS ground up. This is a very hard task, but I would like to program it AFTER building a prototype in another programming language.

Thanks!


-Nirbodha
If you haven't known already, I'm Nirbodha.

Nirbodha means something along the lines of being dull in Bengali. As such, expect my responses to sound like a blockhead made them. Thanks.

Nirbodha
Posts: 15
Joined: Fri Feb 07, 2020 4:36 am

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 5:13 am

mwrich4 wrote:
Wed Feb 12, 2020 3:54 am
Nirbodha wrote:
Fri Feb 07, 2020 4:39 am
Hello. I received a Raspberry Pi 4 B+, and I was wondering if I could create my own operating system.

Are there any good leads for me to follow to create an OS?
It is best to have a wealth of experience from having used a variety of operating systems before creating your own.

Studying FreeRTOS is the best practical way to learn about coding some of the mechanisms operating systems rely on. Taking a Comp Sci curriculum first is even better.

It is NOT an easy nor quick task. It takes a tremendous familiarity with the host CPU to make an effective OS. For learning, those concerns are minimal.


Would a computer science curriculum like the one in Khan Academy suffice?
If you haven't known already, I'm Nirbodha.

Nirbodha means something along the lines of being dull in Bengali. As such, expect my responses to sound like a blockhead made them. Thanks.

Nirbodha
Posts: 15
Joined: Fri Feb 07, 2020 4:36 am

Re: Creating an Operating System on the Raspberry Pi

Thu Feb 13, 2020 5:19 am

Heater wrote:
Fri Feb 07, 2020 8:28 am
Nirbodha ,
Are there any good leads for me to follow to create an OS?
Yes:

1) Learn to program in Rust: https://www.rust-lang.org/

2) Write operating system for the Pi: https://github.com/rust-embedded/rust-r ... -tutorials

You might want to checkout some books on OS design:
https://www.amazon.com/Design-UNIX-Oper ... 0132017997
https://www.amazon.com/Modern-Operating ... 013359162X
Thank you!
If you haven't known already, I'm Nirbodha.

Nirbodha means something along the lines of being dull in Bengali. As such, expect my responses to sound like a blockhead made them. Thanks.

Return to “Beginners”