Page 1 of 2

Firmware question?

Posted: Fri Mar 25, 2016 2:06 pm
by k2tom
Hi,

My question is whether the Raspberry Pi firmware has changed (over the past year or so) SUCH THAT is assumes/requires some (newer?) structure of the kernel/kernel7.img file. I was always under the impression that - by default - the firmware loaded the IMG file to 0x8000 (in the ARM's address space), then it takes the ARM out of reset and jumps there (0x8000). Is there something more now?

Now the background...
I have RPis = RPiB, RPiA, RPiA+, RPi2B, RPi0 and (as of Wednesday) RPi3B. (But let's forget about the 3B for the time being.)

My kernel.img and kernel7.img files are really just bootloaders for the ARM. They use the UART for the user interface. As far as the ARM2835/2836 hardware, they configure the IC (interrupt controller), UART (TBE and RCA interrupts), TIMER, SysTimer, and GPIO (to assign the USART RXD0/TXD0 pins). The ONLY difference between the kernel.img and the kernel7.img file is the register base (for 2835 it's 0x20000000 and for the 2836 it's 0x3f000000).

This worked fine for more than a year (on the B, A, A+ and 2B) (i.e., since the 2B came out). Then came the RPi 0... and I had to update my firmware to firmware the knows about the RPi 0. Once I did that.. the 2836 2B doesn't work anymore! So I have OLD firmware that (again, forgetting the 3B for the moment) works on the B/A/A+/2B (but NOT the 0).. and I have NEW firmware that works on the B/A/A+/0 (but NOT the 2B).

Any thoughts/comments?

Another curiosity.. In my ARM bootloader (the kernel/kernel7.img), I DON'T setup the caches. And again, the 2835 B/A/A+ code is EXACTLY the same as the 2836 2B code (except for the register base). But - for "compute-bound" tasks - the 900MHz 2836/2B takes almost exactly 3.6 times LONGER than the 700MHz 2835. Again, any thoughts?

Regards,

Tom

Re: Firmware question?

Posted: Fri Mar 25, 2016 4:06 pm
by k2tom
A related post set me on the right path (thanks RST!), namely that the newer firmware starts the 2B up in the HYP mode. So I'm trying (so far, unsuccessfully) to switch to SVC (or SYS) mode on startup.

Re: Firmware question?

Posted: Sat Mar 26, 2016 1:30 am
by Ultibo
The HYP mode issue keeps coming up so there are a few posts about it on this forum, sometimes the same answer in a different way can help to make this clearer.

If you have a look at this file

https://github.com/ultibohub/Core/blob/ ... otrpi2.pas

it contains the code for both the HYP to SVC mode switch and the switch from non secure to secure SVC mode. Even though Ultibo core is written in pascal this code is entirely standard ARM assembler inside pascal functions (you can only do the switch in assembler). This code works on both RPi2B and RPi3B and with either the old or new firmware version.

The HYP to SVC mode switch is basically the same code used by Linux and by rst in Circle, the non secure to secure switch is not really official but relies on the fact the arm loader in the firmware cannot enable the MMU or page tables.

If you intend to start up the secondary cores then each of them also has to do the switch from HYP to SVC mode as well when you release them, this is not shown in the file above because it is done elsewhere.

Re: Firmware question?

Posted: Sat Mar 26, 2016 12:45 pm
by dwelch67
Okay so this finally worked for me! There is some magic to touching the exceptions that made the difference. Normally it just hangs the arm.

had to add this to the front to make the difference between working and not. without a need for handlers I just set the stack pointer and bl notmain

Code: Select all

.globl _start
_start:
    ldr pc,reset_handler
    ldr pc,undefined_handler
    ldr pc,swi_handler
    ldr pc,prefetch_handler
    ldr pc,data_handler
    ldr pc,unused_handler
    ldr pc,irq_handler
    ldr pc,fiq_handler
reset_handler:      .word reset
undefined_handler:  .word hang
swi_handler:        .word hang
prefetch_handler:   .word hang
data_handler:       .word hang
unused_handler:     .word hang
irq_handler:        .word hang
fiq_handler:        .word hang

reset:
    mov r0,#0x8000
    mov r1,#0x0000
    ldmia r0!,{r2,r3,r4,r5,r6,r7,r8,r9}
    stmia r1!,{r2,r3,r4,r5,r6,r7,r8,r9}
    ldmia r0!,{r2,r3,r4,r5,r6,r7,r8,r9}
    stmia r1!,{r2,r3,r4,r5,r6,r7,r8,r9}

    mov sp,#0x8000
    mov r0,pc
    bl notmain
hang: b hang
then using this

Code: Select all

.globl GETCPSR
GETCPSR:
    mrs r0,cpsr
    bx lr

.globl StartupSwitch
StartupSwitch:
    mrs r0, cpsr
    eor r0, r0, #0x1A
    tst r0, #0x1F
    bic r0, r0, #0x1F
    orr r0, r0, #0xD3 ;@ #0x80 | 0x40 | 0x13
    bne .LNoSwitch
    orr r0, r0, #0x100
    msr spsr_cxsf, r0
    .word 0xE12EF30E  ;@msr ELR_hyp, lr  (Not supported by the FPC compiler)
    .word 0xE160006E  ;@eret             (Not supported by the FPC compiler)
.LNoSwitch:
    msr cpsr_c, r0
    bx lr
I do this

Code: Select all

    hexstring(GETCPSR());
    StartupSwitch();
    hexstring(GETCPSR());
and get this

200001DA
200001D3

very interesting that not touching memory at 0x0000 it hangs, but if it were to actually run those exceptions it should hit my hang infinite loop and/or reset again and reprint the stuff it had printed to that point and/or go into an infinite loop of doing that.

Very curious. Thanks for the push though, claiming success meant I should be able to as well. Can I steal this code?

David

Re: Firmware question?

Posted: Sat Mar 26, 2016 12:47 pm
by dwelch67
Also I didnt realize you wrote this in pascal. I have tried using that toolchain as a cross compiler for microcontrollers and has marginal success that later turned to failure with some update. I might have to revisit it...

Are you cross compiling with it or running it natively on an arm?

Re: Firmware question?

Posted: Sat Mar 26, 2016 12:53 pm
by dwelch67
BSD does it as well

Code: Select all

	mrs	r0, cpsr
	and	r0, r0, #(PSR_MODE)	/* Mode is in the low 5 bits of CPSR */
	teq	r0, #(PSR_HYP32_MODE)	/* Hyp Mode? */
	bne	1f

	/* Ensure that IRQ, and FIQ will be disabled after eret */
	mrs	r0, cpsr
	bic	r0, r0, #(PSR_MODE)
	orr	r0, r0, #(PSR_SVC32_MODE)
	orr	r0, r0, #(I32_bit | F32_bit)
	msr	spsr_cxsf, r0
	/* Exit hypervisor mode */
	adr	lr, 2f
	msr	elr_hyp, lr
	eret
#endif

Re: Firmware question?

Posted: Sat Mar 26, 2016 1:18 pm
by dwelch67
Ahh, sorry it IS resetting the arm...

Code: Select all

12345678 
00008064 
200001DA <- gets and prints cpsr then does switch 
212345678  <- starts to print cpsr, but resets
00008064 
200001D3 <- now both print outs of the cpsr are in svc mode
800001D3 


.globl GETCPSR
GETCPSR:
    mrs r0,cpsr
    bx lr

.globl StartupSwitch
StartupSwitch:
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bxne lr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf, r0
    .word 0xE12EF30E  ;@msr ELR_hyp, lr
    .word 0xE160006E  ;@eret
    msr cpsr_c, r0
    bx lr

int notmain ( unsigned int earlypc )
{
    unsigned int ra;

    hexstring(0x12345678);
    hexstring(earlypc);
    hexstring(GETCPSR());
    StartupSwitch();
    hexstring(GETCPSR());

    return(0);
}
first time I tried it it didnt do that

Code: Select all

12345678                                                                                         
00008064                                                                                         
200001DA                                                                                         
200001D3 
very interesting. doesnt matter which subtle flavor of StartupSwitch I use, yours or mine or something in between. Not sure how it worked for me a few times before without noticing this reset.

Sorry that I hijacked someone elses thread...

Re: Firmware question?

Posted: Sat Mar 26, 2016 1:44 pm
by k2tom
dwelch67,

No worries about "hijacking" the thread. I've learned a lot. This is truly one of those "I didn't know what I didn't know" moments. Going back a year, when I first got my RPi2B, I spent a few minutes looking at the bare metal threads at the time... changed the register base from 0x20000000 to 0x3f000000... and WOW! I thought: it can't be this easy! But it was! Though, of course, if I had waited a few months and had the "newer" firmware from the beginning... I probably would've gone half-insane.

Thanks,

Tom

Re: Firmware question?

Posted: Sat Mar 26, 2016 2:02 pm
by dwelch67
okay some code snippets. It is clean now. Not exactly sure what fixed it or maybe it still isnt completely fixed and this is just luck. I moved the other three cores first then switched core 0 to SVC. The other three should not have been affected by the vector table changing they should be nowhere near that area by the time. Geez this was with a bootloader, and human speed typing to load the program. no way were they messing around the vector table they were in their loop waiting to be pointed to something to run.

Code: Select all

.globl one_wait
one_wait:
    b .

.globl two_wait
two_wait:
    b .

.globl three_wait
three_wait:
    b .



.globl StartupSwitch
StartupSwitch:
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bxne lr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf, r0
    .word 0xE12EF30E  ;@msr ELR_hyp,lr
    .word 0xE160006E  ;@eret
    msr cpsr_c, r0
    bx lr


    hexstring(0x12345678);
    hexstring(earlypc);
    PUT32(0x4000009C,(unsigned int )one_wait);
    PUT32(0x400000AC,(unsigned int )two_wait);
    PUT32(0x400000BC,(unsigned int )three_wait);
    hexstring(0xAAAAAAAA); //kill some time, do I need this?
    for(ra=0x0000;ra<0x1000;ra+=4) PUT32(ra,0xEAFFFFFE);
    hexstring(0xBBBBBBBB); 
    hexstring(GETCPSR());
    StartupSwitch();
    hexstring(GETCPSR());
    hexstring(0xCCCCCCCC); 


output

Code: Select all

12345678 
0000800C 
AAAAAAAA 
BBBBBBBB 
200001DA 
200001D3 
CCCCCCCC 
Now I just need to have the other three cores cleanly leave hyp mode. Which at this point shouldnt be a big deal. filling the beginning of ram with a branch to self, implies that no events happened reset or otherwise.

Re: Firmware question?

Posted: Sat Mar 26, 2016 2:15 pm
by dwelch67
yep, dumb luck, broken again...

Re: Firmware question?

Posted: Sun Mar 27, 2016 2:01 am
by Ultibo
Hi David,

Since so many people seem to start with your examples I think it would be great if you could get the handling of this included. This HYP mode issue seems to be a bit of a roadblock for many trying out bare metal on the Pi.

I reread your posts a few times to try and understand what you are seeing, I know for sure that I'm not seeing the reset behavior you are because my reset vector points to code that simply loops blinking the LED.

My best guess is that the call to StartupSwitch from notmain is blowing up on return because of some expectation by the compiler about the register state, I would bet that if you changed your _start function to call the startup switch before notmain then it would work

Code: Select all

.globl _start
_start:
   
    bl StartupSwitch

    mov sp,#0x8000
    mov r0,pc
    bl notmain
hang: b hang
You should not need to setup the vectors or release the secondary cores before doing the HYP to SVC switch, it should work as the first (or almost first) call during startup. If you want to save the mode before the switch to later print as debug then store the contents of CPSR to a known address like 0x2000 or somewhere before the call.

Also note that the Linux/BSD variants of the switch code are written as inlined functions so they load the LR for the return based on a label whereas in my code I use an actual function call so the return is the LR from the call.

Code: Select all

   //from Linux or BSD switch code
   adr   lr, 2f
   msr   elr_hyp, lr
Just one to watch out for really, use one or the other method but don't mix them up.

Cheers.

Re: Firmware question?

Posted: Sun Mar 27, 2016 7:59 am
by AlfredJingle
Hi ultibo:

Many thanks for your code for switching to the secure state! Based on it I added the functionality to my project and it works.

I have a question though: Why the cache-cleaning and barrier-instructions? This early in the boot-proces the memory is still strongly ordered, which, as far as I understand it with my limited experience, this means that it should work without cache-cleaning and barrier-instructions. At least it does in my project. What am I missing?

Re: Firmware question?

Posted: Sun Mar 27, 2016 9:50 am
by Ultibo
Hi AlfredJingle,

The primary reason for the cache clean, invalidate, barrier sequence is because I couldn't get the behavior to work predictably without it, sometimes it would work sometimes not depending on the timing. We know from the snippet of code released on pastebin (see the header of my file) that the instruction cache has been enabled by the firmware loader.

I suspect that the important bit is the invalidate instruction cache, flush branch target cache plus barriers to ensure ordering. Somewhere in the ARM documentation it shows this sequence as the correct way to write code to memory and then execute it so I included the whole sequence to be certain it works every time.

Glad you got it to work in your project as well!

Cheers.

Re: Firmware question?

Posted: Sun Mar 27, 2016 2:19 pm
by dwelch67
Ultibo,

Thanks for the comments, I will try some more on this I had given up for the moment.

It had dawned on me perhaps the problem was trying to use it as a function as both the stack pointer and link register are specific to mode, so the lr and sp going into the function call not the ones coming out. But that didnt help when I put this code (without bx lr, just a skip if not in that mode, actually I should have just taken that check out as I know what mode I am in) it simply hung. I am not using any config.txt, perhaps other folks are? I prefer a solution that works without a config.txt.

Re: Firmware question?

Posted: Sun Mar 27, 2016 3:10 pm
by dwelch67
to make it simpler assembly only. with the skip I get blinking leds, without not...

Code: Select all

.globl _start
_start:
    ;@ b skip
    mrs r0,cpsr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf, r0
    msr ELR_hyp,lr ;@.word 0xE12EF30E  ;@msr ELR_hyp,lr
    eret           ;@.word 0xE160006E  ;@eret
    msr cpsr_c, r0
skip:
    ldr r0,=0x3F200000
    ldr r2,=0x00008000
    ldr r3,=0x00000008

    ldr r1,[r0,#0x10]
    bic r1,r1,#0x00E00000
    orr r1,r1,#0x00200000
    str r1,[r0,#0x10]

    ldr r1,[r0,#0x0C]
    bic r1,r1,#0x00038000
    orr r1,r1,#0x00008000
    str r1,[r0,#0x0C]

top:
    str r2,[r0,#0x20]
    str r3,[r0,#0x2C]
    bl delay
    str r2,[r0,#0x2C]
    str r3,[r0,#0x20]
    bl delay
    b top

delay:
    mov r4,#0x18000
dloop:
    subs r4,r4,#1
    bne dloop
    bx lr

Code: Select all

	arm-none-eabi-as -mcpu=cortex-a7 -march=armv7a hype.s -o hype.o
	arm-none-eabi-ld hype.o -Ttext=0x8000 -o hype.elf
	arm-none-eabi-objdump -D hype.elf > hype.list
	arm-none-eabi-objcopy hype.elf -O binary hype.bin
	arm-none-eabi-objcopy hype.elf -O ihex hype.hex
I guess if you were using a config.txt it would be to avoid going into hyp mode in the first place and we wouldnt be having this discussion.

Code: Select all

00008000 <_start>:
    8000:	e10f0000 	mrs	r0, CPSR
    8004:	e3c0001f 	bic	r0, r0, #31
    8008:	e3800013 	orr	r0, r0, #19
    800c:	e16ff000 	msr	SPSR_fsxc, r0
    8010:	e12ef30e 	msr	ELR_hyp, lr
    8014:	e160006e 	eret
    8018:	e121f000 	msr	CPSR_c, r0
doesnt matter if I use my bootloader or copy the bin file to the sd card. no blinking.

Re: Firmware question?

Posted: Sun Mar 27, 2016 3:35 pm
by dwelch67
I see your comment about the lr. It should have worked for my function. For _start code though lr is the wrong register. msr ELR_hyp,pc didnt work so I use a regular register.

Code: Select all

    mrs r0,cpsr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    add r3,pc,#4
    msr ELR_hyp,r3 
    eret          
tried this in the C based program

Code: Select all

    mov r2,lr
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bxne lr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    add r3,pc,#4
    .word 0xe12ef303  ;@ msr    ELR_hyp, r3
    .word 0xe160006e  ;@ eret
    bx r2
still hangs. So will just go with a _start solution and see what happens.

Re: Firmware question?

Posted: Sun Mar 27, 2016 3:42 pm
by dwelch67
My current solution, going to run with this until it breaks...

Code: Select all

.globl _start
_start:
    ;@ b skip
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bne skip
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    add r3,pc,#4
    .word 0xe12ef303 ;@ msr ELR_hyp,r3
    .word 0xe160006e ;@ eret
skip:

Re: Firmware question?

Posted: Sun Mar 27, 2016 4:43 pm
by dwelch67
Ahh, right sp is banked, so it has to be repaired for switching back. So using lr still doesnt make sense for ELR_hyp since you need to be in svc mode to change the svc_sp yes? hmm...

Code: Select all

.globl StartupSwitch
StartupSwitch:
    mov r2,lr
    mov r3,sp
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bxne lr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    mov r0,sp
    add r3,pc,#4
    .word 0xe12ef303  ;@ msr    ELR_hyp,r3
    .word 0xe160006e  ;@ eret
    mov sp,r0
    bx r2
So another way to deal with the banked registers. Results were not consistent though, using the .bin version as kernel7.bin is consistent so there may be an issue using the bootloader. The bootloader doesnt do anything that would cause an issue, so that doesnt make sense.

Going to just run with switching first thing...and see how that goes...

Code: Select all

.globl StartupSwitch
StartupSwitch:
    mrs r0,cpsr
    and r1,r0,#0x1F
    cmp r1,#0x1A
    bxne lr
    msr SP_svc,sp
    msr LR_svc,lr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    msr ELR_hyp,lr
    eret

Re: Firmware question?

Posted: Sun Mar 27, 2016 6:37 pm
by dwelch67
so tell me the rest of the story? It is in SVC mode as far as the CPSR indicates. But I cannot access the VBAR, nor SCR. You have some code that calls the SCM secure monitor? with a value of 0x00000001? is that relevant to this? I was hoping to move the VBAR to 0x8000 rather than setup an exception table at 0x0000 for interrupt handling. Setting up the handlers at 0x0000 work so the example works but feels like I am not completely back into a non-HYP mode.

Thanks for the help!

David

Re: Firmware question?

Posted: Mon Mar 28, 2016 8:45 pm
by AlfredJingle
Hi David,

After leaving HYP mode you end up in non-secure SVC mode. You cannot change the VBAR while non-secure, nor access SCR.
The workaround is to use a SCM monitor call, which uses the code at memory 0x8 to jump to the secure monitor code. This code than switches the security state to secure. Only after returning from the secure monitor are you in secure SVC mode giving you, among other goodies, access to SCR and VBAR.

The number after SCM is immaterial for the function of SMC, it allows the calling function to give a message to the secure monitor.

Re: Firmware question?

Posted: Mon Mar 28, 2016 9:07 pm
by AlfredJingle
Hi Ultimo,

Thanks for your reply. I was just trying to understand. I only started programming assembly last August and basically spend the last 3 weeks or so fighting to get my project ( a direct subroutine threaded Forth compiler) to run on a RasPi3. The main stumbling block turned out to be dirty caches. The pi3 apparently is more sensitive to dirty caches than the pi2.
Anyhow that was why I was interested. I solved the problem with the dirty caches on my pi3 and it now all seems to run stable. And thanks to your code now also in secure mode!

Re: Firmware question?

Posted: Tue Mar 29, 2016 12:00 am
by dwelch67
Okay I see what you are doing. I thought the caches were off. Hmm...that explains a lot! My bootloader is creating a cache coherency problem, no wonder I was having issues. If I load a program then wait a few seconds, then allow it to run, it behaves differently than if I load and run immediately, even though that is still human speed hitting a key after the download finishes. I changed the bootloader to supposedly disable the caches, but that still doesnt quite do it. You still have to wait. I should have the bootloader invalidate everything and see if that helps.

So my prior question was some confusion and your answer is now confusing. With only changing to svc mode, it appears I CAN change the vbar for interrupts. I had the HVBAR instruction in there that I cut and pasted over and missed a number, that wasnt working for me for SVC mode. But VBAR does....for interrupts... I though I was crashing in svc mode simply reading the vbar, but again it was reading hvbar. Then looking at your secure code it starts by reading the vbar, which confused me, thought that crashes. So since I was able to modify the vbar for interrupt handling purposes, I tried for this, nope doesnt work.

Then I assumed we had the caches off, nope, they are on, so all the steps you have seem to be required.

Code: Select all

.globl _start
_start:
    ldr pc,reset_handler
    ldr pc,undefined_handler
    ldr pc,swi_handler
    ldr pc,prefetch_handler
    ldr pc,data_handler
    ldr pc,hyp_handler
    ldr pc,irq_handler
    ldr pc,fiq_handler
reset_handler:      .word reset
undefined_handler:  .word hang
swi_handler:        .word smc
prefetch_handler:   .word hang
data_handler:       .word hang
hyp_handler:        .word hang
irq_handler:        .word hang
fiq_handler:        .word hang

reset:
    ;@ b skip
    mrs r0,cpsr
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr spsr_cxsf,r0
    add r0,pc,#4
    msr ELR_hyp,r0
    eret
skip:

    mrc p15, 0, r1, c12, c0, 0 ;@ get vbar
    mov r0,#0x8000
    ;@ mov r1,#0x0000
    ldmia r0!,{r2,r3,r4,r5,r6,r7,r8,r9}
    stmia r1!,{r2,r3,r4,r5,r6,r7,r8,r9}
    ldmia r0!,{r2,r3,r4,r5,r6,r7,r8,r9}
    stmia r1!,{r2,r3,r4,r5,r6,r7,r8,r9}

    ;@ Clean Data Cache MVA
    mov r12,#0
    mcr p15, 0, r12, c7, c10, 1
    ;@ Perform a data synchronisation barrier
    dsb
    ;@ Invalidate Instruction Cache
    mov r12, #0
    mcr p15, 0, r12, c7, c5, 0
    ;@ Flush Branch Target Cache
    mov r12, #0
    mcr p15, 0, r12, c7, c5, 6
    ;@ Perform a data synchronisation barrier
    dsb
    ;@ Perform an instruction synchronisation barrier
    isb
    ;@ Perform a secure monitor call (Not supported by the FPC compiler)
    smc #0

    ;@ stop caching
    mrc p15,0,r2,c1,c0,0
    bic r2,#0x1000
    bic r2,#0x0004
    mcr p15,0,r2,c1,c0,0

    mov sp,#0x8000
    mov r0,pc
    bl notmain
hang: b hang

smc:
    mrc p15, 0, r1, c1, c1, 0
    bic r1, r1, #1
    mcr p15, 0, r1, c1, c1, 0
    movs    pc, lr
And now I can read the scr without crashing! So thanks for this as well! Now to disable the caches in the bootloader.

Re: Firmware question?

Posted: Tue Mar 29, 2016 1:00 am
by dwelch67
Am I seeing this right, on the pi3 in aarch32 mode (no config.txt), they do NOT turn on the caches. but on the pi2 (no config.txt) they do?

Thanks for all the help and for starting this thread for me to hijack...

David

Re: Firmware question?

Posted: Tue Mar 29, 2016 2:38 am
by Ultibo
Hi David,

Looks like you've got it now!

The combination of the caching being on and your bootloader loading code into memory to execute does explain a lot about the odd behavior you were seeing.

Just to clarify the situation with the VBAR, it is a banked register so you can read/write it in both secure and non secure modes but which copy you see will depend on the value of the NS bit in the SCR.
Am I seeing this right, on the pi3 in aarch32 mode (no config.txt), they do NOT turn on the caches. but on the pi2 (no config.txt) they do?
I haven't ever checked, can't see any reason why that would have changed though.

Re: Firmware question?

Posted: Wed Mar 30, 2016 2:48 pm
by swarren
I'm curious why running in HYP mode is any kind of problem? It works fine for the Linux kernel (in fact it very quickly drops to EL1).

Rather than attempt to switch from HYP mode back up to EL3, why not simply avoid switching into HYP mode in the first place. Simply use "kernel_old=1" in config.txt, link your code to be loaded at address 0, and then you're in complete control of the code that runs on the ARM. The Pi Foundation has even published their 32-bit ARM startup code, so assuming there are no license incompatibilities with your project, you could derive your code from that. See popcornmix's first comment at https://github.com/raspberrypi/firmware/issues/579.