User avatar
DougieLawson
Posts: 41189
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Writing an OS from scratch

Sun Apr 18, 2021 10:22 am

shacharshemesh wrote:
Sun Apr 18, 2021 10:02 am

Or, you can acknowledge that I'm only after creating the operating system from scratch. Not the hardware. Not the compiler. Just the operating system.
So we're back to https://wiki.osdev.org/Beginner_Mistakes
Best of luck. See you in thirty years time when you have something that may be usuable.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

swampdog
Posts: 616
Joined: Fri Dec 04, 2015 11:22 am

Re: Writing an OS from scratch

Sun Apr 18, 2021 10:32 pm

shacharshemesh wrote:
Sun Apr 18, 2021 10:02 am
Or, you can acknowledge that I'm only after creating the operating system from scratch. Not the hardware. Not the compiler. Just the operating system.
This is the point you seem to be missing. If you create the OS from scratch, the code created by gcc won't work. You'll have to (at least) write your own linker then patch gcc to work with it.

User avatar
Gavinmc42
Posts: 5480
Joined: Wed Aug 28, 2013 3:31 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 12:00 am

If you create the OS from scratch, the code created by gcc won't work. You'll have to (at least) write your own linker then patch gcc to work with it.
So make the compiler first before the OS?

Oberon is one of the few OS's out there with built in Compiler.
Any others?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ejolson
Posts: 7084
Joined: Tue Mar 18, 2014 11:47 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 12:47 am

DougieLawson wrote:
Sun Apr 18, 2021 10:22 am
shacharshemesh wrote:
Sun Apr 18, 2021 10:02 am
Or, you can acknowledge that I'm only after creating the operating system from scratch. Not the hardware. Not the compiler. Just the operating system.
So we're back to https://wiki.osdev.org/Beginner_Mistakes
Best of luck. See you in thirty years time when you have something that may be usuable.
For the record Linus started on Linux early 1991 and announced the project in August of the same year with
Linus wrote: I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready.
By 1992 Linux was running a full version of the X Window System.

https://en.wikipedia.org/wiki/History_of_Linux

My understanding is that it took a little longer for Brian Kernighan and Dennis Ritchie to take their PDP-7 prototype and turn it into a practical operating system running on the PDP-11. That's because they needed to write all the assemblers and compilers as well.

The L4 micro kernel and it's predecessor were also both conceived and implemented by the single person Jochen Liedtke.

The port of Linux to the Raspberry Pi did not take millions of hours, people, pounds or dollars to reach a usable state.

Andrew Tanenbaum wrote Minix from scratch to use in his teaching.

From one point of view, an operating system is not nearly as complicated as many other pieces of software, for example, the codes which implement the weather forecasting models used by NCAR or the multi-physics simulations used behind the fence at LANL. Moreover, the existence of so much open-source code has made writing an operating system even easier than before.

In my opinion, the reason most operating system projects fail is not because it is very difficult, but because most people decide it's very boring. In the end they try out a few ideas and then decide Linux or Windows is good enough.

The Omega2 looks like it could be a good platform to target. Every time something like this comes up, however, I suggest it would be nice to create a work-alike of the operating system which ran of the first Cray supercomputers. That hardware was well documented and emulators exist, but both the source code and binaries for the operating system have been lost to the ravages of time.

swampdog
Posts: 616
Joined: Fri Dec 04, 2015 11:22 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 1:32 am

Gavinmc42 wrote:
Mon Apr 19, 2021 12:00 am
If you create the OS from scratch, the code created by gcc won't work. You'll have to (at least) write your own linker then patch gcc to work with it.
So make the compiler first before the OS?
Actually you don't need to. It's a common misconception the gcc compiler is everything. It isn't. It may appear that way and things may have been integrated over the years but there are three distinct parts.

a) headers - description of the system capabilities
b) compiler - translation of description into binary
c) binaries - implementation of system capabilities

The one thing you don't actually need to change (much) is gcc. Just drive it differently like they used to do in the old days of makefiles: cpp -> cc -> ld and it's only 'ld' which needs major change because it will not be the linux OS underneath. You also have to rewrite a lot of other (c) stuff like 'nm' and 'ar' and lets just assume this new OS isn't going to handle dynamic libs.

A couple of examples: (mno-)cygwin uses gcc but produces windoze binaries & aix gcc typically sits on top of the ibm linker. It's "binutils" where the work needs to go.

Silly thought: We don't actually know what OS the OP desired to write from scratch. For all we know they may intend to just load a python interpreter as a kernel.

shacharshemesh
Posts: 15
Joined: Wed Dec 11, 2019 7:34 am
Contact: Website

Re: Writing an OS from scratch

Mon Apr 19, 2021 2:22 am

swampdog wrote:
Sun Apr 18, 2021 10:32 pm
This is the point you seem to be missing. If you create the OS from scratch, the code created by gcc won't work. You'll have to (at least) write your own linker then patch gcc to work with it.
I'm always amazed by people confidently telling me something which I have already done in the past is impossible.

Here's the bootstrapping sequence:
  • Compile a dedicated version of gcc for the new platform. There numerous HOWTOs on the internet, and since they stopped requiring compiling all the requirements separately it's not even particularly complicated. Just give it the correct CPU and specify the OS as "none".
  • Create a linker script for the platform. This is the "porting" part of the project. You use the platform's memory map to decide which address gets what. Typically, the text segment (code) and RODATA goes in ROM, the BSS segment goes in RAM, and the DATA segment goes in RAM, but stored in ROM.
  • Write a loader that copies the data segment from ROM to RAM, initializes the BSS segment to all zeros, and starts execution at the "main" function (or wherever else you decide).
  • Use the resulting toolchain to compile and link the executable you write. This results in an ELF file.
  • Use objcopy to create a raw file from your ELF file.
  • Burn that raw file to ROM
  • Profit
Feel free to explain to me, in great details, why the above doesn't work. Just bear in mind, when you do, that I've done this before. Aside from the loader, you do not need to actually write code for any of those steps (unless you consider the linker script "code"). The standard toolchain is perfectly capable of all of the above.

If you want to know more, I'll be more than happy to post a link to my Youtube channel once I get my Omega2 and set it up. Dougie will by, then, have more than 70 years of experience (the 40 he already has + the 30 it will take me to get there), and will not need it, but the rest of you might find it useful.

Which is the whole point of the exercise.
swampdog wrote: We don't actually know what OS the OP desired to write from scratch.
I want to start a video YouTube channel showing how to get from a SoC and datasheets to a running operating system. The OS doesn't need to be particularly functional. It just needs to showcase the major elements of modern operating systems.
Likes to engage in lost causes and doomed projects.
Author and developer of the Practical programming language (https://practical-pl.org)

The longer you spend arguing online with a fool, the greater the chances he's doing the same thing.

User avatar
Gavinmc42
Posts: 5480
Joined: Wed Aug 28, 2013 3:31 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 2:50 am

showcase the major elements of modern operating systems.
What would that look like for an IoT embedded OS?
The IoT field is going to get even bigger and the OS for that will be different than HID desktop/handheld things that come with attached slow human.

Considering that seems to be were the Omega is aimed at anyway.
IoT OS's would be more interesting for me.
Task switcher to full RTOS?
Standalone to connected to distributed OS's.

About the only one I have tried was FreeRTOS.
I make single purpose gadgets with Pi's, but a structured approach would be better than my adhoc versions.
Some were standalone, some had HID, none are autonomous yet.

Robot OS are of Interest too.
ROS is huge and not easy to learn on.

My definition of OS is a bit more lax than those that regard Linux as "THE" typical OS.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

swampdog
Posts: 616
Joined: Fri Dec 04, 2015 11:22 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:07 am

shacharshemesh wrote:
Mon Apr 19, 2021 2:22 am
swampdog wrote:
Sun Apr 18, 2021 10:32 pm
This is the point you seem to be missing. If you create the OS from scratch, the code created by gcc won't work. You'll have to (at least) write your own linker then patch gcc to work with it.
I'm always amazed by people confidently telling me something which I have already done in the past is impossible.

Here's the bootstrapping sequence:
  • Compile a dedicated version of gcc for the new platform. There numerous HOWTOs on the internet, and since they stopped requiring compiling all the requirements separately it's not even particularly complicated. Just give it the correct CPU and specify the OS as "none".
  • Create a linker script for the platform. This is the "porting" part of the project. You use the platform's memory map to decide which address gets what. Typically, the text segment (code) and RODATA goes in ROM, the BSS segment goes in RAM, and the DATA segment goes in RAM, but stored in ROM.
  • Write a loader that copies the data segment from ROM to RAM, initializes the BSS segment to all zeros, and starts execution at the "main" function (or wherever else you decide).
  • Use the resulting toolchain to compile and link the executable you write. This results in an ELF file.
  • Use objcopy to create a raw file from your ELF file.
  • Burn that raw file to ROM
  • Profit
That's not an operating system. That's "bare metal". There's a forum for that here.
shacharshemesh wrote:
Mon Apr 19, 2021 2:22 am
Feel free to explain to me, in great details, why the above doesn't work. Just bear in mind, when you do, that I've done this before. Aside from the loader, you do not need to actually write code for any of those steps (unless you consider the linker script "code"). The standard toolchain is perfectly capable of all of the above.

If you want to know more, I'll be more than happy to post a link to my Youtube channel once I get my Omega2 and set it up. Dougie will by, then, have more than 70 years of experience (the 40 he already has + the 30 it will take me to get there), and will not need it, but the rest of you might find it useful.

Which is the whole point of the exercise.
An OS at minimum defines an input/output interface and hopefully other services. It might even multi-task. That API needs to be written by you and then needs its own compiler/interpreter running inside that OS. Ideally the result needs to be capable of building itself.
shacharshemesh wrote:
Mon Apr 19, 2021 2:22 am
swampdog wrote: We don't actually know what OS the OP desired to write from scratch.
I want to start a video YouTube channel showing how to get from a SoC and datasheets to a running operating system. The OS doesn't need to be particularly functional. It just needs to showcase the major elements of modern operating systems.
The Omega2 looks interesting so I glanced at it. It uses OpenWRT & you'll be completely replacing that as well?

ejolson
Posts: 7084
Joined: Tue Mar 18, 2014 11:47 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:25 am

swampdog wrote:
Mon Apr 19, 2021 4:07 am
The Omega2 looks interesting so I glanced at it. It uses OpenWRT & you'll be completely replacing that as well?
It seems like the Omega2 and particularly the Omega2+ are out of stock where I checked. Is this temporary or will the devices never be easily available?

Maybe targeting the Pi B+ or Zero would be a better idea.
Last edited by ejolson on Mon Apr 19, 2021 6:31 am, edited 1 time in total.

shacharshemesh
Posts: 15
Joined: Wed Dec 11, 2019 7:34 am
Contact: Website

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:29 am

swampdog wrote:
Mon Apr 19, 2021 4:07 am
That's not an operating system. That's "bare metal". There's a forum for that here.
The OS runs on the bare metal.
swampdog wrote:
Mon Apr 19, 2021 4:07 am

An OS at minimum defines an input/output interface and hopefully other services. It might even multi-task. That API needs to be written by you
Yes. That's the whole point of the exercise.
swampdog wrote:
Mon Apr 19, 2021 4:07 am
and then needs its own compiler/interpreter running inside that OS. Ideally the result needs to be capable of building itself.
No. An operating system has no such requirement, and lots of operating systems do not live up to it.

The purpose of this exercise is educational. Getting gcc or clang to run on this new OS is not cost effective in that regard, and will, therefor, likely not happen. It will run applications, but those will have to be cross-compiled.

Like I said elsewhere in this thread, when the focus switches to user space, I'll likely switch to Linux.
swampdog wrote:
Mon Apr 19, 2021 4:07 am
The Omega2 looks interesting so I glanced at it. It uses OpenWRT & you'll be completely replacing that as well?
Not "as well". The whole point is to replace it.

OpenWRT is just a Linux distribution. Linux is an operating system.

What I'm not yet sure about is the boot loader. IIRC, the Omega is using u-boot. There is some educational value in showing how to write one of those, but I'll have to make sure I, and more importantly anyone following along, do not brick the device in the process.
Likes to engage in lost causes and doomed projects.
Author and developer of the Practical programming language (https://practical-pl.org)

The longer you spend arguing online with a fool, the greater the chances he's doing the same thing.

cleverca22
Posts: 3587
Joined: Sat Aug 18, 2012 2:33 pm

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:33 am

shacharshemesh wrote:
Mon Apr 19, 2021 4:29 am
swampdog wrote:
Mon Apr 19, 2021 4:07 am
and then needs its own compiler/interpreter running inside that OS. Ideally the result needs to be capable of building itself.
No. An operating system has no such requirement, and lots of operating systems do not live up to it.
i would argue that this is the critical difference behind a dumb single-goal kernel, and an OS

an OS is a collection of binaries, that can be loaded at runtime, and that loading requires a libc and the linker/compiler to know about your kernel
but a dumb kernel, can just lack a loader, all code is linked into the kernel at build time, and it just boots and does some task, and has no flexibility

but something like micropython blurs the lines more

shacharshemesh
Posts: 15
Joined: Wed Dec 11, 2019 7:34 am
Contact: Website

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:35 am

ejolson wrote:
Mon Apr 19, 2021 4:25 am
swampdog wrote:
Mon Apr 19, 2021 4:07 am
The Omega2 looks interesting so I glanced at it. It uses OpenWRT & you'll be completely replacing that as well?
It seems like the Omega2 and particularly the Omega2+ are out of stock where I checked. It's this temporary or will the devices never be easily available?
I've seen places where it was out of stock, but going through the Omega site redirected me to DigiKey, where I had no problem buying it with immediate availability. They also directed to Mouser, where some of the expansion components I wanted were out of stock, but had a few days of lead time. This leads me to believe any stock problems are temporary.
ejolson wrote:
Mon Apr 19, 2021 4:25 am
Maybe targeting the Pi B+ or Zero would be a better idea.
The Pi was my first preference. See the beginning of this topic for why I to abandoned it.
Likes to engage in lost causes and doomed projects.
Author and developer of the Practical programming language (https://practical-pl.org)

The longer you spend arguing online with a fool, the greater the chances he's doing the same thing.

shacharshemesh
Posts: 15
Joined: Wed Dec 11, 2019 7:34 am
Contact: Website

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:38 am

cleverca22 wrote:
Mon Apr 19, 2021 4:33 am
shacharshemesh wrote:
Mon Apr 19, 2021 4:29 am
swampdog wrote:
Mon Apr 19, 2021 4:07 am
and then needs its own compiler/interpreter running inside that OS. Ideally the result needs to be capable of building itself.
No. An operating system has no such requirement, and lots of operating systems do not live up to it.
i would argue that this is the critical difference behind a dumb single-goal kernel, and an OS

an OS is a collection of binaries, that can be loaded at runtime, and that loading requires a libc and the linker/compiler to know about your kernel
but a dumb kernel, can just lack a loader, all code is linked into the kernel at build time, and it just boots and does some task, and has no flexibility

but something like micropython blurs the lines more
There is a huge difference between being able to run a dynamic loader and being able to run a complete toolchain. Just to illustrate this point: you can install Linux without installing either a compiler or a linker, but that doesn't stop you from running binaries that require dynamic linking.
Likes to engage in lost causes and doomed projects.
Author and developer of the Practical programming language (https://practical-pl.org)

The longer you spend arguing online with a fool, the greater the chances he's doing the same thing.

cleverca22
Posts: 3587
Joined: Sat Aug 18, 2012 2:33 pm

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:48 am

ive been mostly working on a fork of LK: https://github.com/librerpi/lk-overlay

my goal wasnt to make an entire kernel from scratch, but to make something usable, and to make good examples of how to drive the various hardware within the rpi

currently, it has drivers for:
  • the HVS, which is basically hardware accelerated sprites
  • the VEC, needed to generate composite video
  • sdhost, to manage an SD card
  • the ddr2 controller of the pi0-pi3 range, allowing
  • the arm control block, allowing execution of arm code
its able to compile into many forms:
  • bootcode.bin mode, runs at the very first stage on a pi0-pi3 model, and can optionally bring dram online and load a static .elf file without any relocation support
  • start.elf mode, 2nd stage, loaded by either bootcode.bin build of LK or the official bootcode.bin, less limitations on the binary size
  • kernel.img mode, runs instead on the arm core, with help from the official firmware, and can partially manage an hdmi display
there is no dynamic linker support, just blind chainloading of another kernel

User avatar
Gavinmc42
Posts: 5480
Joined: Wed Aug 28, 2013 3:31 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 5:11 am

Every ones idea of "OS" is different, but I need to be able to do things.

In the last few years a real "OS" has been something I can run an app on.
That app for me must be made on the same OS.
Plus standalone without external tools.

The closest I got was compiling Lil to run on Ultibo.
http://runtimeterror.com/tech/lil/
The next step would have been add an editor to write Lil apps on this "OS".

I also extended the Ushell utilities that came with Ultibo to access the hardware from a command line.
But there is no "shell" interpreter like Bash, yet.

OS's by themselves do nothing useful, applications run in/on the OS.
Linux come with Shell scripting, I use that all the time to make apps.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

LTolledo
Posts: 5140
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Writing an OS from scratch

Mon Apr 19, 2021 9:35 am

As the OP claimed to have "done it before"....
I suggest the OP just proceed anyway with the method the OP knows... and not "waste time" answering to post here...
that way the OP can proceed smoothly with the OP's project.

once the OP has made the "first step" in the procedure the OP outlined.... the OP may share "excerpts" of the process (some codes perhaps, and some outputs) for us to know the status.

....and just proceed along the next step....and so on and so forth.... until the goal is reached.

if the OP does run into some roadblocks along the way.... the OP may ask for advise, hints, tips from forum members who have first-hand experience in doing the same...

the best of luck to the OP's team....
..and hope to see some results...later on.... on youtube (or any video sharing sites that exist in that time frame)
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

User avatar
Gavinmc42
Posts: 5480
Joined: Wed Aug 28, 2013 3:31 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 11:55 am

Just spent a few hours playing with my latest toy, CM4 and I/O board.
For some of the uses a CM4 could be used for a full Linux OS is not needed.
Mind you 32 different versions to pick from.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ejolson
Posts: 7084
Joined: Tue Mar 18, 2014 11:47 am

Re: Writing an OS from scratch

Mon Apr 19, 2021 4:03 pm

shacharshemesh wrote:
Mon Apr 19, 2021 2:22 am
I want to start a video YouTube channel showing how to get from a SoC and datasheets to a running operating system. The OS doesn't need to be particularly functional. It just needs to showcase the major elements of modern operating systems.
Will you write most of the OS using the Practical programming language or something else?

shacharshemesh
Posts: 15
Joined: Wed Dec 11, 2019 7:34 am
Contact: Website

Re: Writing an OS from scratch

Tue Apr 20, 2021 4:35 am

ejolson wrote:
Mon Apr 19, 2021 4:03 pm
Will you write most of the OS using the Practical programming language or something else?
I actually considered it. Sadly, Practical isn't nearly mature enough for that task, so the question of how educational that would be is moot.
Likes to engage in lost causes and doomed projects.
Author and developer of the Practical programming language (https://practical-pl.org)

The longer you spend arguing online with a fool, the greater the chances he's doing the same thing.

Return to “Advanced users”