dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Firmware question?

Wed Mar 30, 2016 2:59 pm

Because its there. To know how to do it.

Likewise the knowledge of staying in HYP mode and using it in that mode.

Personally I prefer not to use config.txt if at all possible, I had to for aarch64, but perhaps that is only temporary? My guess is well over 99% of the pis out there do not use a config.txt, so basically it is barely tested. Likewise they have already done this in the past and there is no reason for them to have any compatibility from one firmware release to another with respect to config.txt hacking, the loyalty is clearly not with the bare metal folks (less than one percenters) but with the target use case. So personally the most important way to do bare metal is to only replace kernel(7).img, including undoing something in software rather than using config.txt. Then config.txt modes are secondary, esp now that we have these two new quad cores, more stuff to learn.

Bare metal programming is a very personal thing just like our favorite text editor, keyboard, mouse, political party, and religion. We are not forced into one option fortunately on this platform so to each his/her own...Those are my personal feelings on it, there are many folks here that prefer the config.txt approach and that is just fine.

So I wanted to know how to do this, and after enough people had said they had it working, I dug in...

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Firmware question?

Thu Mar 31, 2016 6:26 pm

Yes, the educational aspect makes sense.

I was thinking more from a "production SW" perspective; anything one might want to share with other people and have it "just work" reliably for them. I haven't looked at the code anyone is using to do this yet. If you're resetting the CPU to get back out of HYP, I imagine that would be reliable. However, if you're invoking SMC to get back into the (essentially non-existen) secure monitor in the "ARM stub" provided by the VC FW, having patched that up to do something you want, I'd expect that to fail if the ARM stub code is re-organized to fix issues or add features, since the code will move around, and the vector addresses may change.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Firmware question?

Fri Apr 01, 2016 2:10 pm

I have no interest in giving out fish, if anything it is a teach them to fish. They have to take it from there, esp if they are going for production software. But production software starts with baby steps, as does ad hoc. Getting that first instruction running, getting C bootstrapped, seeing if the program is running (blink an led). Scary stuff for most people their first times, too many to count ways things could go wrong.

Up to the reader/developer to harden their own products. I have no interest in doing that for them, I already have a day job...Now in my day job I may come across a cortex-a7/8 and need to know how to get in or out of hyp mode for debugging or other reasons. So now I have a taste of that in my back pocket.

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: Firmware question?

Mon Apr 04, 2016 7:31 am

Hi swarren:

Running in hyp-mode is only a problem if you want to use or change any of the security registers, like: SCR, SDER, NSACR etc. These registers can only be changed from a secure world. See page 4219 of the ARMv8 Architecture Reference Manual. If you do not need to use the security registers than hyp-mode is probably fine.
LINUX leaves hyp-mode as the first action of the boot-process. It than switches to EL3 to configure the security registers.
It is a misunderstanding that a reset is needed for going to monitor-mode. You set a vector to your monitor routine at dress 0x8, clean caches if needed, and call the monitor routine with SMC. The monitor routine can be hugely complex but also can be very short, the Monitor-routine I use has exactly 3 opcodes. See page 4554 of the Reference Manual.

Code: Select all

MONITOR:

movw r9, 0x0130
mcr p15, 0, r9, c1, c1, 0
movs pc, lr
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Firmware question?

Tue Apr 05, 2016 2:07 am

AlfredJingle wrote:It is a misunderstanding that a reset is needed for going to monitor-mode.
Within the context I was talking about, that's not true.

In general, code running at a privilege level less than the secure monitor cannot write to the memory containing the code of the higher privilege levels. It's an unusual quirk of the Pi SW stack that the EL3/monitor-mode SW doesn't take steps to protect itself, because it isn't implementing any security features. In a more typical SW stack, you wouldn't be able to write to address 8 to modify what the SVC call did. My comments were intended to highlight the lack of generality of any method other than performing a CPU reset to change the monitor mode SW.

Also note that the base address of the monitor mode exception vectors is SW-programmable. It's quite possible that the Pi Foundaiton could change the "ARM stub" code and find they need to re-arrange it to fit for some reason, and hence change the exception base. The same technique for replacing the monitor code would still work in this case, it's just that the address of the code you need to replace would be different. I believe that typically code running outside secure mode can't read the register containing the monitor exception vector base register, so I don't think you could write code to automatically adapt to such a change, but I'm not 100% sure on this point.

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Firmware question?

Tue Apr 05, 2016 1:59 pm

Am I hearing that the operating systems. Linux, BSD, etc leave HYP mode as a first thing? If so why are the pis being switched into HYP mode to just go back out? Or am I misunderstanding?

rst
Posts: 404
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: Firmware question?

Tue Apr 05, 2016 8:14 pm

dwelch67 wrote:Am I hearing that the operating systems. Linux, BSD, etc leave HYP mode as a first thing? If so why are the pis being switched into HYP mode to just go back out? Or am I misunderstanding?
Because it's possible to use a hypervisor that way. Some people wanted to do so: https://github.com/raspberrypi/firmware/issues/369

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Firmware question?

Wed Apr 06, 2016 12:16 am

Hmmm, they could have just provided a patch or a special raspian or something for the folks that wanted that. The needs of the few out weight ...

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Firmware question?

Wed Apr 06, 2016 2:13 am

On ARM64 the exception levels are named a bit more sanely, so I'll mainly use that terminology, but I believe everything below applies equally to ARM32 if you substitute the correct CPU mode names etc.

The CPU boots in EL3 (~SVC). This typically runs a secure monitor. Hypervisors run in EL2 (HYP). Any OS kernel typically runs in EL1 (system). User-space applications run in EL0 (user). All ARM32 versions of the Linux kernel recommend the kernel be entered in EL2 mode, but do allow SVC instead. All ARM64 versions of the Linux kernel require the CPU be entered in either EL2/HYP (recommended) or non-secure EL1; EL3 is not allowed.

Thus, the ARM stub switching to HYP mode isn't something that's there just to benefit a few unusual people, but rather is done to comply with the Linux booting rules/recommendations.

If you're writing some other OS for the Pi, it's best to fit into the standard ARM exception level usage, and run in EL1/system mode. That should work on any CPU/SoC and interact well with any other parts of the SW stack. It's quite unusual to run an OS (rather than a secure monitor, or secure OS on ARM32) in SVC/EL3.

rst
Posts: 404
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: Firmware question?

Wed Apr 06, 2016 5:31 am

swarren wrote:Thus, the ARM stub switching to HYP mode isn't something that's there just to benefit a few unusual people, but rather is done to comply with the Linux booting rules/recommendations.
Once you are in EL1 there is no way to get back into higher exception levels without the support of the software running there. So it's not unusual to boot in a higher EL. I didn't want to tell this.
If you're writing some other OS for the Pi, it's best to fit into the standard ARM exception level usage, and run in EL1/system mode. That should work on any CPU/SoC and interact well with any other parts of the SW stack. It's quite unusual to run an OS (rather than a secure monitor, or secure OS on ARM32) in SVC/EL3.
My AArch64 port of the Circle bare metal environment runs in EL1t (main thread, using sp_el0) and in EL1h (exception handlers, using sp_el1). Currently it expects to be booted in EL3. This has to be changed so that EL1, 2 or 3 are allowed.

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: Firmware question?

Wed Apr 06, 2016 11:11 am

to: dwelch

An OS has to leave hyp-mode as soon as possible after booting in order to set up the security registers and security related vectors and traps and what have you. This can only be done from a secure world. And hyp-mode is unsecure by definition. I have no clue why the pi is switched to hyp-mode. Just keeping it in secure supervisor mode seems to be easier for all.
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

dwelch67
Posts: 955
Joined: Sat May 26, 2012 5:32 pm

Re: Firmware question?

Thu Apr 07, 2016 12:51 am

Thanks.

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Firmware question?

Thu Apr 07, 2016 1:59 am

AlfredJingle wrote:to: dwelch

An OS has to leave hyp-mode as soon as possible after booting in order to set up the security registers and security related vectors and traps and what have you. This can only be done from a secure world. And hyp-mode is unsecure by definition. I have no clue why the pi is switched to hyp-mode. Just keeping it in secure supervisor mode seems to be easier for all.
That's not quite true.

Registers that can only be set up in secure mode should be set up by code running in secure mode prior to switching to less-privileged HYP/EL2 (or non-secure system-mode/EL1) to boot the kernel. That's how the architecture is designed to work.

Once you've entered HYP/EL2, there's no way back to a more privileged state except by:
  • Resetting the CPU (which will lose all kinds of state)
  • Making an SMC call into SVC/EL3 mode. In a secure system, this can only invoke explicitly supported APIs that SVC/EL3 deliberately exports.
  • Exploiting some security hole (or a deliberate lack of security) in the secure code. It sounds like this is what people are doing on the Pi to get SVC mode to do what they want; over-writing the code at the secure exception vectors with their own code and then invoking SVC.
OSs entered at anything above (more privleged than) system-mode/EL1 typically do switch out of that mode as soon as possible, but switch to (less-privileged) system-mode/EL1. That's the typical mode/exception-level an OS kernel runs at.

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: Firmware question?

Thu Apr 07, 2016 9:23 am

To: swarren,

This being the bare metal forum, isn't that what we are talking about? The earliest bootcode? Nobody is using a hole in security, as security is simply not yet setup this early in the boot process. That is why setting the vector for a smc call is still possible. The issue is not a hole in security, the issue is that somehow Raspberry has decided to start the CPU in unsecure hyp-mode rather than a standard secure supervisor mode. This makes it necessary to do the trick with the vector in order to reach a secure mode and thus finally being able to setup security. As far as I know every Linux (and indeed any serious OS) running on the Raspberry has to do the same trick in one way or another in the early boot-process ór use the kernel_old=true setting in the config.sys.
Anyhow, I am thankful that ultibo has described a simple solution on getting in the secure world during boot. It works great and it was just what was needed.
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Return to “Bare metal, Assembly language”