Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Spider-OS a new operating system

Fri Jan 25, 2019 6:05 pm

Hi everyone. I introduce myself. I am developing a new operating system entirely written in assembly language : https://spider-os.com. A real passionate challenge ;) , in the idea of ​​understanding the different components of the Raspberry Pi.

The goal is to create a home automation system to control the many heterogeneous systems that appear every day.

Here I am at the beginning, and I update my website to show the progress of my development. Do not hesitate to leave me comments. See you soon.

User avatar
DarkPlatinum
Posts: 793
Joined: Thu Nov 02, 2017 2:30 pm
Location: Unknown
Contact: Website

Re: Spider-OS a new operating system

Fri Jan 25, 2019 7:08 pm

Pretty cool! Why did you decide to program it in assembly rather than a different language? For your website, did you purchase your own domain? What about making your website slightly more fancier? If you would like to do that I can help you get started. (I'm running a website on a Pi. You can see how it looks by clicking the icon next to my name)
1 * Raspberry Pi Zero W, 1 * Raspberry Pi 2, 1 * Raspberry Pi 3 1 * Raspberry Pi 3B + :mrgreen:

Check Out My Raspberry Site (Run on a Raspberry Pi 3B :) ): Https://Html.dynu.net

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Fri Jan 25, 2019 8:21 pm

Hi DarkPlatinum,

it is true the assembly seems limited because it does not bring all the possibilities of a high level language like Python. But we know exactly the code that is generated. An assembly instruction is a code directly executed by the machine: we can not do more optimized. And this is necessary when you want to be at the ready of the hardware when designing a driver such as USB, or management of processor interrupts. C is a low-level language, so close too, but I dismissed it because the programming environment is heavy, and requires compilation.
Nothing is easier with FASMARM https://arm.flatassembler.net/. We modify its code, we compile in a few seconds, and we can send it on the Raspberry for testing.
Moreover with a little rigor, I manage to program in a structured way. And finally we can do a lot of things similar to C, such as structures.

Concerning the domain spider-os.com, yes indeed I own.
The website is light, I recognize it. But I must say that I also learn HTML, PHP ... in parallel. But I think I improve it over time, in order to learn many more things.
Your website is nice, thank you for your proposal, but for the moment I prefer to improve it by myself.

User avatar
DarkPlatinum
Posts: 793
Joined: Thu Nov 02, 2017 2:30 pm
Location: Unknown
Contact: Website

Re: Spider-OS a new operating system

Fri Jan 25, 2019 9:49 pm

Aran wrote:
Fri Jan 25, 2019 8:21 pm

The website is light, I recognize it. But I must say that I also learn HTML, PHP ... in parallel. But I think I improve it over time, in order to learn many more things.
The great thing about having a light website is that its fast. Learning HTML and PHP is a great way using a website. When will we see a release for you OS?
1 * Raspberry Pi Zero W, 1 * Raspberry Pi 2, 1 * Raspberry Pi 3 1 * Raspberry Pi 3B + :mrgreen:

Check Out My Raspberry Site (Run on a Raspberry Pi 3B :) ): Https://Html.dynu.net

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Sat Jan 26, 2019 9:08 am

Yes I like everything that is fast ;-)
It will take a few more months, before having an operational system. But I would put some screenshots or videos online, when it will be really interesting.

User avatar
Gavinmc42
Posts: 2882
Joined: Wed Aug 28, 2013 3:31 am

Re: Spider-OS a new operating system

Sat Jan 26, 2019 9:45 am

This will be an Aarch64 OS?
I am coming around to microkernel thinking these days,the Linux kernel is just so big.
A smaller one I might be able to learn ;)

I am also learning about Agent Model languages like Pony,
They should lend themselves to multicore and distributed computing.
Not sure how you would do that is assembly?

If you like RF then do Bluetooth LE, just the Uart terminal will be a good place to start.
Lots of baremetal startup code has been done now, so it is a good time to do this by following in their footprints.

Because so many languages are using LLVM, I had thought about using LLVM IR as the programing language ;)
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

bzt
Posts: 343
Joined: Sat Oct 14, 2017 9:57 pm

Re: Spider-OS a new operating system

Sat Jan 26, 2019 11:49 am

Hi,

First of all, sounds great! I really would like to see a demo and an online repo!
Aran wrote:
Fri Jan 25, 2019 8:21 pm
it is true the assembly seems limited because it does not bring all the possibilities of a high level language like Python. But we know exactly the code that is generated. An assembly instruction is a code directly executed by the machine: we can not do more optimized.
Well, I think you meant that, and we must stress "seems" :-) Although you're wrong about optimisation, because compilers can do a really good job with high level languages (just check out gcc's optimisation flags). To my experience one has to be a really skilled programmer to come up with a better Assembly code than the generated one (but it is not impossible). The main reason for this is that compilers do micro-optimisations, while a talented and experienced programmer can also do algortihmic and macro-optimisations. Unfortunately most programmers never learned how to do that.
And this is necessary when you want to be at the ready of the hardware when designing a driver such as USB, or management of processor interrupts. C is a low-level language, so close too, but I dismissed it because the programming environment is heavy, and requires compilation.
I think you're right, but for the wrong reason. Assembly also needs compilation, and you can write an USB driver in C. Interrupt handlers are trickier, it is often more benefitial to have a small Assembly wrapper than using obscure C attributes like "interrupt" or "register".
Nothing is easier with FASMARM https://arm.flatassembler.net/. We modify its code, we compile in a few seconds, and we can send it on the Raspberry for testing.
Now that's a unarguable reason :-) Fasm is the best Assembler ever, period. (And I say that with more than 30 years of experience with masm, tasm, nasm and gas).
Moreover with a little rigor, I manage to program in a structured way. And finally we can do a lot of things similar to C, such as structures.
And much much more. This will blow up your mind: Object Oriented Assembly. Fasm can do all of that stuff and more, check out "virtual" directive and it's friends too.

In case you're in a need of a small, simple, efficient disassembler for your project, may I suggest udisasm. I know it's a C header file, but you can generate an Assembly version of it easily with "gcc -S".

If you don't mind an invitation, it would be great to see you and your OS on http://forum.osdev.org. More RPi OSes are very welcome, right now the community is a bit IBM PC oriented.

Cheers,
bzt

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Sat Jan 26, 2019 12:31 pm

Hi Gavinmc42,
Well, I'm learning a lot of terms with you ;-)

At my level, since I started from scratch, the kernel is really very simple. I started working on multitask, which works thanks to interruptions caused by the timer. But I reinvent everything, so I do not know the technologies you mention.
Regarding the RF, actually Bluetooth low energy is a track, but I'm also thinking Zigbee, or inexpensive wifi component: the ESP 8266.
Since I want to create a command interpreter, I think about a language. I would like it to be close to the hardware, like assembly, and high level like Python: an unlikely equation ;-)

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Sat Jan 26, 2019 12:54 pm

Hi bzt,
glad to see your enthusiasm on my project.
I take note of your remarks. I know the virtual directive, which I use for pointing to structures. Yes indeed I am amazed to see that we can do object-oriented assembly :-)
I thought to be the only extraterrestrial to dare to program assembly : happy to see that this is not the case ;-)
Thank you for the invitation, I will probably go when time allows me.

User avatar
Gavinmc42
Posts: 2882
Joined: Wed Aug 28, 2013 3:31 am

Re: Spider-OS a new operating system

Sat Jan 26, 2019 10:27 pm

Hi Gavinmc42,
Well, I'm learning a lot of terms with you ;-)
Yep, like I learn a lot from bzt and the others who have done baremetal.

The really great thing with Pi's is the ability to try all these things and with the support here and elsewhere you are not alone.
Most of my day to day stuff these days is done on Ultibo, not exactly baremetal or OS, some sort of weird, pick anything mix.

If I was going to pick assembly it would be for the Aarch64, mainly because it is supposed to be simpler and more power efficient.
Compilers are variable, results are in a long post on the Off Topic thread about Basic
That Object Assembly looks very interesting if very brief.
And FASM with ARM will get added to my Pi Gentoo64 OS ;)

FPGA's are now cheap and interesting enough to grab my attention.
A Lattice UP5K fpga can fit a RISC-V cpu, but suppose you could design your own cpu, what architecture/assembler/compiler would you use?
Learning assembler helps learn the cpu architecture.

Latttice have some interesting BNN/CNN (neural network) apps, but you don't need 32bit cpu's to handle 1,8,16 bit NN data.
What cpu's do you use and how to program them and yet allow for FPGA hardware that varies depending on application.
What Language/OS can handle this flexibility.

Moore law limit was reached about 10 years ago, CPU's won't get much faster, so they stick more cpu's on.
But adding more cpu cores does not scale well, so now they are adding fpga's to the chips.
Intel acquired Altera, wonder that that will produce?

We can play with all this stuff using Pi's.
In my lifetime I have gone from using 1MHz 8bit CPU's to 4 core 64bit 1GHz+ cpu's.
CPU's may not make it to 10GHz, so traditional ways need reconsidering.
Youngsters using Pi's now are the next generation of coders, they won't have the luxury we had of "there will be 2x faster cpu's next year."
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

dwelch67
Posts: 954
Joined: Sat May 26, 2012 5:32 pm

Re: Spider-OS a new operating system

Tue Jan 29, 2019 6:09 pm

Note that writing an operating system in assembly does not mean small nor fast. This are misunderstandings, doesnt meant you are learning the system any better than you would using mostly C plus a little ASM. That is also a misunderstanding. Doesnt mean you shouldnt try but if those are your goals and this is your first time to do this then expect to not meet those goals.

If this is just a really really hard way to make your controller, but a really really fun way. then go for it.

I highly recommend the pi-zero first, but either way these platforms are very good for this. The documentation is poor but despite that the resources in this forum and in open source examples is really second to none, I have not seen anything else with this kind of of a community outside avrfreaks but those devices/boards are/were simpler...

Have fun.
David

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Tue Jan 29, 2019 8:22 pm

Hi David,

currently Spider-OS only takes 60 Kbytes with all the features I mentioned. I think it's pretty small ;-)
As far as speed is concerned, it depends mainly on how to program. For that I test the performances and I modify the code accordingly.
I think that with assembly, you can easily program any chip by writing its registers. Yes you'll tell me the C also with the pointers, but I find that the assembly is more direct.
The downside is that it takes a lot of time. Fortunately it's a hobby, and I have patience.
Indeed, I have already been inspired by many contributions on the Raspberry forums, and on the Internet. Among the many that I could mention, there is Peter Lemon who did a nice job on the framebuffer and the Videocore.
I look forward to introducing you to an operational system, but it will not be for many months.

code_exec
Posts: 271
Joined: Sun Sep 30, 2018 12:25 pm

Re: Spider-OS a new operating system

Thu Jan 31, 2019 6:00 pm

Suggestions:

1. Make OS ARM64 to take full advantage of the Pi 3's SoC.
2. Include basic stuff such as a web browser, notes program, calculator, calendar, and clock.

Does this OS support the built-in WiFi provided by the Pi 3?
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

User avatar
Gavinmc42
Posts: 2882
Joined: Wed Aug 28, 2013 3:31 am

Re: Spider-OS a new operating system

Fri Feb 01, 2019 2:26 am

Web Browser?
Er I don't call that basic stuff, that's the Everest of coding.

I would be interested if OpenVG works as that paves the way to SVG and CSS.
Making that the Everest base camp?
Very strange but there is not much around in OpenVG code examples.
Just reasons why OpenGLES is better?

Peter Lemon showed the way for baremetal Openxx.

WiFi requires cyrpto support, big ask to do that in assembler.
Bluetooth LE on the other hand is just a fancy uart ;)
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

LdB
Posts: 1100
Joined: Wed Dec 07, 2016 2:29 pm

Re: Spider-OS a new operating system

Fri Feb 01, 2019 3:36 pm

OpenVG was too late to the party and only does 2D so it had little interest.

The wikipedia entry tells the story and it's end
https://en.wikipedia.org/wiki/OpenVG
On September, 2011 OpenVG working group decided not to make any regular meeting for further standardization. However, working group decided to continue maintenance and promotion of OpenVG 1.1 specification.
.
Nothing has happened on the spec in 7 years tells a story

User avatar
Gavinmc42
Posts: 2882
Joined: Wed Aug 28, 2013 3:31 am

Re: Spider-OS a new operating system

Fri Feb 01, 2019 11:33 pm

[Nothing has happened on the spec in 7 years tells a story/quote]
No need to change, it's perfect as it is ;)
For 2D GUI it is ideal, simpler than OpenGLES.
I whipped up a LCARS style interface in a few hours just using rects, rounded rects, circles and text.
On baremetal :D

Simple touch screen interfaces for single purpose use, perfectly fine.
Auto-scales on displays so no counting pixels.
I am now busy collecting Orville screenshots as I like the colour scheme.
Have to learn the path and curve stuff to do proper lists that are more optimized.

People forget the VC4 is an old design so code that was designed at the same time is the proper code.
Of course if it had a VC5 then Vulkan would be my choice.
Can Vulkan run on VC4?
For an OS that would be better but would it be fast enough?

So single purpose OS or general purpose OS?
Lots of multipurpose OS's, we don't have as much choice for embedded OS's.
Most are old and not good enough for graphics, that has been changing recently.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

LdB
Posts: 1100
Joined: Wed Dec 07, 2016 2:29 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 4:10 am

VC4 can't do full vulkan it requires GLES3 hardware.

What you can do is as much as possible of the spec as some have tried
https://github.com/Yours3lf/rpi-vk-driver

User avatar
Gavinmc42
Posts: 2882
Joined: Wed Aug 28, 2013 3:31 am

Re: Spider-OS a new operating system

Sat Feb 02, 2019 5:19 am

VC4 can't do full vulkan it requires GLES3 hardware.
Well Pi's only have OpenGLES hardware but I'm running an OpenGL Desktop.

Just because it has not been done does not mean it will not be done.
Hopefully Pi4-5-6-etc are Vulkan hardware capable.

Right now I am busy installing compilers on Sakaki's dual Debian.
GCC 6.3, Clang 3.9., Rustc 1.24, FPC 3.0.0, two versions of each, aarch64 and aarch32.
I can now compile for 32 and 64 bit Pi's perhaps even at the same time :o
Ok, so they are not the latest versions but I am trying to get my head around this.

Might be old hat in PC and virtual machine world, but this is Pi's, $55 AUD SoC boards.
I could set up a networked Pi just as a compiler box.
Just got Midnight Commander going, both versions, that is my default network file manager and editor.
Yep FP, the text mode IDE for FPC works too.

I think all those compilers have been known to make baremetal code for Pi's, one left?
http://arm.flatassembler.net/
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

LdB
Posts: 1100
Joined: Wed Dec 07, 2016 2:29 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 6:50 am

Gavinmc42 wrote: Well Pi's only have OpenGLES hardware but I'm running an OpenGL Desktop.
With the rest of the spec emulated in software and why the performance is down.
Gavinmc42 wrote:Just because it has not been done does not mean it will not be done..
Yes but almost everyone has deemed it isn't worth doing because it will be slow, so you need someone who has a reason.
Gavinmc42 wrote: GCC 6.3, Clang 3.9., Rustc 1.24, FPC 3.0.0, two versions of each, aarch64 and aarch32.
I can now compile for 32 and 64 bit Pi's perhaps even at the same time :o
Ok, so they are not the latest versions but I am trying to get my head around this.
The CortexA53 is designed to co run 64 bit and 32 bit OS and apps you simply need to start it in 64 bit mode.
Its 50-100 lines of assembler or C to play around with it if you really want to play .. I can setup a sample.
I believe bzt may already have a sample from his challenge he ran (The challenge was to write a small block of assembler that could automatically work out if it was running in 32/64 bit mode at runtime).
Image

There are multiple ways to make a 32/64 bit binary app its trivial the cortexa53 does all the rest.
The alternative is as you said do it with VM but you would only really do that if you wanted the older Pi processors.
Gavinmc42 wrote:I think all those compilers have been known to make baremetal code for Pi's, one left?
http://arm.flatassembler.net/
Almost any C compiler for the Pi can act as an assembler along with all a battery of GNU assemblers themselves there is no shortage of options. You lost me with the significance of flatassembler. On any GCC 64 bit compiler then "gcc -m32-bit" will force it to compile 32 bit code
which works so long as you aren't using any prebuilt libraries. So the need for 2 compilers can be obselete but at worst you need both.

Anyhow this is all getting a bit wide for this thread as the OP has shown no indication he wants to do this sort of thing but it's relatively easy if he did.

code_exec
Posts: 271
Joined: Sun Sep 30, 2018 12:25 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 9:46 am

Gavinmc42 wrote:
Sat Feb 02, 2019 5:19 am
Well Pi's only have OpenGLES hardware but I'm running an OpenGL Desktop.
The Pi 2 and 3 do support OpenGL through an experimental driver. The driver is far from production-ready, but it does provide hardware acceleration through the VC4 GPU.
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

Aran
Posts: 21
Joined: Fri Jan 25, 2019 5:55 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 11:00 am

Indeed before creating the elements that interact with the user as a web browser, you must already have the basic components of the system. I published an article about this, which lists all the essential basic modules : here.
But it must be said that a calculator is a must have, which is easy to make compared to the implementation of the wifi protocol ;-)

bzt
Posts: 343
Joined: Sat Oct 14, 2017 9:57 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 12:41 pm

Hi,
Aran wrote:
Sat Feb 02, 2019 11:00 am
Indeed before creating the elements that interact with the user as a web browser, you must already have the basic components of the system. I published an article about this, which lists all the essential basic modules : here.
But it must be said that a calculator is a must have, which is easy to make compared to the implementation of the wifi protocol ;-)
Nice list, but I think you forgot two important items:

1. Hardware Abstraction Layer and Device Drivers
Although it is possible to hardwire all low level drivers in a kernel, it is not advisable. It's a much better practice to create a common interface, and implement independent drivers using that interface for each device (regardless if you're using monolithic or microkernel architecture). Your source will became much maintainable, and easier to port or expand with new drivers.

2. User Space Library
Again, although it is possible to directly use your kernel's syscall interface, it is really a bad practice. It's much better to create a user space library which hides all the details, and your user space applications should be linked with that library calling the kernel functions as normal functions. In ARM terms this means that your library should have wrapper functions around the SVC instruction, which you can call with a simple BL instruction from your applications using the normal ABI.

Cheers,
bzt

LdB
Posts: 1100
Joined: Wed Dec 07, 2016 2:29 pm

Re: Spider-OS a new operating system

Sat Feb 02, 2019 4:08 pm

While I definitely agree with bzt on point 1, on point 2 I am going to suggest that is very linux and single core centric answer.

However it is not worth getting into until you start writing a multicore kernel and there are a number of elegant alternatives.
Basically you will need your system call load to be balanced across all the cores so such a simple system call mechanism can lead to other problems. You will also require communication between the cores as some of the kernel code must be sequential even though running in parallel on different cores. Anyhow that is for way down the track.

For now I would simply say you need another abstraction layer as point 2 and leave it at that.

bzt
Posts: 343
Joined: Sat Oct 14, 2017 9:57 pm

Re: Spider-OS a new operating system

Mon Feb 04, 2019 12:31 am

LdB wrote:
Sat Feb 02, 2019 4:08 pm
While I definitely agree with bzt on point 1, on point 2 I am going to suggest that is very linux and single core centric answer.

However it is not worth getting into until you start writing a multicore kernel and there are a number of elegant alternatives.
Basically you will need your system call load to be balanced across all the cores so such a simple system call mechanism can lead to other problems. You will also require communication between the cores as some of the kernel code must be sequential even though running in parallel on different cores. Anyhow that is for way down the track.
I think I may have been unclear a bit. I'm not suggesting a full-blown libc, just a minimal wrapper library one can link (or with fasm more like include) and use as "BL fopen" instead of "SVC 0x1234" for example.
For now I would simply say you need another abstraction layer as point 2 and leave it at that.
Yeah, something like that. TBH since fasm is so capable with macros, one could create "spideros_fopen" mnemonic as well in that header, it does not need to be a classic library. Good point! "Another abstraction layer" is the proper phrase.

Cheers,
bzt

Heater
Posts: 12095
Joined: Tue Jul 17, 2012 3:02 pm

Re: Spider-OS a new operating system

Mon Feb 04, 2019 8:37 am

Aran,

I'm all for learning assembler. I think all programmers should do so. It's gives one a good appreciation of what goes on "under the hood" in computers. Besides it's interesting and fun.

I also admire your audacity in taking on such a large and complex project in assembler. In days gone by I wrote a lot of code in assembler, even a multi-tasking protected mode OS kernel for 32 bit Intel. I can appreciate the effort involved.

But I have to make some comments:
An assembly instruction is a code directly executed by the machine: we can not do more optimized.
I don't believe this is true. Because:

1) Unless you know the details of of your processor very well it's quite likely a C compiler will do better. As an example: The Intel x86 processor has a means of automatically repeating instructions, intended for copying memory blocks, string comparison and so on, compilers do not use this hardware feature because it turns out to be slower! There are many other such examples for all processor architectures.

2) If you write for 32 bit ARM your code will not be optimal for 64 bit. And vice versa. If it's written in C a simple recompile handles that.

3) It's much easier to explore options, different way to code the same thing, to find an optimal sweet spot if you use C rather than assembler.

4) Change processor architecture and the performance of assembler drops to zero. How will your OS run when the Raspberry Pi becomes a RISC V machine?
this is necessary when you want to be at the ready of the hardware when designing a driver such as USB, or management of processor interrupts.
This is clearly not true. As demonstrated by the fact that pretty much all operating systems are written in a high level language. Only a very small sprinkling of assembler is required in odd corners.
C is a low-level language, so close too, but I dismissed it because the programming environment is heavy, and requires compilation.
C requires compilation. Assembler requires assembling. Yes perhaps a C compilation takes a bit more time but really, it's hardly anything. And think of all that optimization it is doing for you.
I thought to be the only extraterrestrial to dare to program assembly : happy to see that this is not the case
The most fun I had programming in assembler for decades was for the Propeller MCU from Parallax Inc. 32 bits and 8 cores. Sweet. But with limitations that make assembler essential.

Now, I'm not suggesting C is the be all and end all. We have been traveling that path for decades now. Perhaps it's time for a rethink. Like this guy writing his OS in Rust: https://github.com/redox-os/redox The idea there being to make use of the Rust language to achieve a more stable, less buggy and importantly secure OS whilst maintaining the performance of C/assembler.

@Gavinmc42,
Compilers are variable, results are in a long post on the Off Topic thread about Basic
Yes compilers are variable but that thread shows that variability between languages and even small algorithm changes have a much bigger effect on performance than specific C compilers. My take away from that thread is that it's amazing how good they are!

Return to “Bare metal, Assembly language”