Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

[solved] RPi-3 (aarch32) - MMU activation hangs

Sun Feb 11, 2018 6:02 pm

Hi there,

I've been looking into many many source codes that are initializing the MMU for the Raspberry Pi. But what ever I use from them are whatever I fould here in the forum does not see, to work for me.

Whenever it comes to set the M flag in the system control register the PI just hangs and I've currently no clue what the reason could be.

The latest and most simple excample I'm trying to get running is using TTBR0 only with a 1:1 mapping.
Here is what I did so far (based on several excamples grabbed from github or this forum):

Code: Select all

// for MMU config disable D/I cache and MMU in case it has been active before
	asm volatile(
			"mrc p15, #0, r12, c1, c0, #0\r\n" 	// read current SCTRL
			"bic r12, r12, #0x1\r\n"			// switch off MMU bit
			"bic r12, r12, #0x4\r\n"			// switch off D cache bit
			"bic r12, r12, #0x1000\r\n"			// switch off I cache bit
			"mcr p15, #0, r12, c1, c0, #0\r\n"	// set new SCTRL
		);
After this some are writing to the Auxiliry Control Register. However, the ARMv8/Cortex-A53 documentation I have does not indicate that the
SMP parameter in this register exists. And even if I write the value of the examples (1 << 6) to this register, if I read it back the content has not changed.

Code: Select all

asm volatile(
			"mrc p15, #0, r12, c1, c0, #1\r\n"	// read current ACTLR (Auxiliry Control Register)
			"orr r12, r12, #0x40\r\n"			// set bit 6 to enable coherent processor request (ArmV8 docu http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0500g/BABGHIBG.html say's this enable L2 write access to L2ACTLR register)
												// the ARMv7 http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0464f/BABGHIBG.html clearly say's SMP flag at bit 6 for AuxiliryControlRegister :/ can't find any SMP flag in ArmV8 docu
			"mcr p15, #0, r12, c1, c0, #1\r\n"	// set new ACTLR
			"isb\r\n"							// trigger instruction sync barrier
		);
The system control register now reads as:
SCTRL:0xC50838

Now I'm setting the TTBRC like this:

Code: Select all

asm volatile(
			"mcr p15, 0, %0, c2, c0, #0\r\n"	// set TTBRC control register to use TTBR0
			:: "r"(0) :"memory"
		);
Set the TLB base address like this:

Code: Select all

asm volatile(
			"mcr p15, #0, %0, c2, c0, #0\r\n"	// set TTBR0 base address and flags
			: : "r"(((uint32_t)aTLB0) | 3)
		);
The TLB is defined so: static volatile __attribute__((aligned(0x4000))) uint32_t aTLB0[4096]; and already filled with some 1:1 mappings
Finally I try to activate the MMU writing to the system control register like this:

Code: Select all

asm volatile(
			"mrc p15, #0, r12, c1, c0, #0\r\n" 	// read current SCTRL
			"orr r12, r12, #0x1\r\n"			// switch on MMU bit
			"orr r12, r12, #0x4\r\n"			// switch on D cache bit
			"orr r12, r12, #0x1000\r\n"			// switch on I cache bit
			"mcr p15, #0, r12, c1, c0, #0\r\n"	// set new SCTRL
		);
And here the PI hangs... I've setup all interrupt handler but none is called, neither UNDEF_INSTRUCTION nor PREFETCH_ABORT nor DATA_ABORT... the RPI3 just stops continuing.... I would pretty much appriciate if someone might be able to give a hint where to look at or how to analyze further to come to the root cause.

In the past whenever the RPi hangs this was an indicater that some stack has not being setup properly for one of the processor modes, but I do not think that this instruction would cause a CPU mode switch other than the exception modes I've already covered with an interrupt handler...

Thanks in advance for any help.
Schnoogle
Last edited by Schnoogle on Sat Mar 24, 2018 10:09 pm, edited 1 time in total.

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Mon Feb 12, 2018 9:52 pm

Hi,

Don't know if I can help you, but at least I see some issues. Maybe this helps. I only use assembly so you have to do the translation into C yourself.

First: Coherency on a ARMv8 processor is not set using SMP but using the following code.
MRRC p15, 1, r0, r1, c15
orr r0, r0, #64 @ set bit [6] op 1 - this is the non-secure version!!!!
@ as far as I know the secure versie can only be set in MON=el3
MCRR p15, 1, r0, r1, c15 @ Write CPU Extended Control Register = ACTLR
isb

And secondly I set the TTBCR using the following steps:

mov r0, #0x0 @ to use only TTBR0,
mcr p15, 0, r0, c2, c0, 2 @ write a 0 to TTBCR (NOTICE THE 2 AT THE END!!!)
isb

ldr r1, =0x406A @ 0000 0000 0000 0000 0100 0000 0110 1010 shared inner, write back, alocate on write
mcr p15, 0, r1, c2, c0, 0 @ Write to TTBR0[31:0] - non-secure part
isb

As far as I can see the rest of your code seems correct. I can only assume that you set all the flags in the translation table correctly and that you switched successfully from HYP mode to system-mode. Otherwise you have to set the HYP-specific registers, and not the general registers. The ARMv8 Architectural RM gives lots of information on this.
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Fri Feb 16, 2018 11:12 am

Hey there,

thanks for your response. I've tried to implement your suggestions and also changed the whole MMU activation stuff into an ASM routine like this:

Code: Select all

m_init_mmu: //r0 = uTTLBAddr
	mov		r9, r0						// store savely the input parameter
	mov		r0, #0
	bl		ua_sendx					// send r0 content to UART

	mcr		p15, #0, r1, c1, c0, #0		// read the current SCTRL register
	bic		r1, r1, #0x1				// switch off MMU
	bic		r1, r1, #0x4				// switch off D cache
	bic		r1, r1, #0x1000				// switch off I cache
	mcr		p15, #0, r1, c1, c0, #0		// write back SCTRL register

	mov		r0, #1
	bl		ua_sendx					// send r0 content to UART

	mrrc	p15, #1, r1, r2, c15		// read CPU extended control register
	orr		r1, r1, #64					// set SMP flag for inter core coherency
	mcrr	p15, #1, r1, r2, c15		// write CPU extended control register
	isb

	mov		r0, #2
	bl		ua_sendx					// send r0 content to UART

	mov		r1, #0
	mcr		p15, #0, r1, c2, c0, #2		// set TTBR control register to use TTBR0 only
	isb

	mov		r0, #3
	bl		ua_sendx					// send r0 content to UART

	mov		r0, #0x406a
	orr		r9, r9, r0					// build TTLB base address + control flags
	mcr		p15, #0, r9, c2, c0, #0		// set the TTBR0 register
	isb

	mov		r0, #4
	bl		ua_sendx					// send r0 content to UART

	mcr		p15, #0, r1, c1, c0, #0		// read the current SCTRL register
	orr		r1, r1, #0x1				// switch on MMU
	orr		r1, r1, #0x4				// switch on D cache
	orr		r1, r1, #0x1000				// switch on I cache
	mcr		p15, #0, r1, c1, c0, #0		// write back SCTRL register
	isb

	bx	lr
As you might notice I've also put some UART output here and then to track the progress of the code..
And there some interesting thing happens...
UART writes first 0x0 and than 0x1... Well so far so good, but than it took approx. 20 seconds and the UART is displaying 0x3F215040 instead of 2 :?: :?: :?:
Even if I set 2 to be send to UART right before the call it sends a complete weird content...

further more no more stuff is send to UART, the code never reaches the "3" or the "4".
If I leave the code that writes to the CPU extended control register out, than the stuff hangs after sending "4" to UART, mean right when trying to activate the MMU.

The CPU is set from HYP to SYS mode right at the beginning of my code...

Any further hint is very welcome.
Thx in advance.

BR Schnoogle

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Sun Feb 25, 2018 11:26 pm

I looked at your code and I have only two remarks:

a). Did you set a stackpointer? If you switch from HYP to SYS, you have to again define your stack-pointer, as the registers are (partly) mode-specific.

b). I first set the flags for I-cache and D-cache, and in a second action switch on the MMU.

I do a lot of cache-cleaning, purging and invalidating around the initiation, but that is only needed for a warm reset. The rest of your code, apart from the UART-calls, is exactly what I have.


Hope this helps a bit. Happy bug hunting....
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Thu Mar 01, 2018 7:43 pm

Hey there,

thanks for your time looking again at my code.
I'm sure I'm fine in the SYS-mode and I'v properly set all my stack pointers...as all the rest of my code is running fine in SYS mode.

I'm really struggeling with this stuff.
What I found so far, that when calling:

Code: Select all

mrrc	p15, #1, r1, r2, c15
r1 = 0x18000
r2 = 0x51652

But from the ARM doc's bit's "[63:7] -Reserved, res0." this should not be the case!?
However, when zeroing out r1 and r2 and just passing bit 6 set (SMPEN) and calling

Code: Select all

mcrr	p15, #1, r1, r2, c15
my PI hang's for a couple of seconds and than calls the RESET IRQ ....
I've really no clue where to start further analysis...There nothing mentioned that the CPU might stop working when writing the wrong value to this register.
At this moment the caches as well as the MMU is still deactivated ....

Any further idea would be much appreciated....
Thanks in advance
Schnoogle

rst
Posts: 311
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPi-3 (aarch32) - MMU activation hangs

Fri Mar 02, 2018 4:41 pm

I observed two things:
  1. You did not set the Domain Access Control Register (DACR). This register controls the access permission for each memory domain (0-15) used in the translation table entries. Its reset value in UNKNOWN, so it is normally required to set (at least) domain 0 to "Client" (1) or "Manager" (3) before enabling the MMU in SCTLR:

    Code: Select all

    // set Domain Access Control Register (Domain 0 to client)
    asm volatile ("mcr p15, 0, %0, c3, c0,  0" : : "r" (1));
    
  2. If you do not use the "kernel_old=1" entry in config.txt, you do not need to touch the CPU Extended Control Register (mcrr p15, 1, rN, rM, c15), because this has already been done by the ARM stub here.

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Tue Mar 06, 2018 12:58 pm

I literally use the followoing code to switch on SMP:

Code: Select all

	
		MRRC p15, 1, r0, r1, c15
		orr r0, r0, #64				@ set bit [6] op 1 - this is the non-secure version!!!!
							@ the secure versie can only be done in MON=el3											
		MCRR p15, 1, r0, r1, c15		@ Write CPU Extended Control Register = ACTLR
		isb
I do that well before setting the DomainAccess control using:

Code: Select all

		ldr r0, =0xFFFFFFFF			@ set all access domains to "manager" ie to 'no-checks done'
		mcr	p15, 0, r0, c3, c0, 0		@ Write to DACR
		isb
My feeling still is that there is a problem with a stack-pointer somewhere. Especially as your UART writes 0x3F215040. This is the address of the AUX_MU_IO_REG register (=Mini Uart I/O Data). The fact that this address ends up as output from the UART to me seems to point to a stack-pointer problem. This is also based on the long struggle I had myself with getting the pointers correct!
Another option to explore might be to purge/invalidate the caches, translation table and branch-prediction. I do that several times during the setup-fase. And although I cannot explain why but since I do that the RPi starts and restarts 100% reliable.
A last point is that ARM has now published a document on initiating a Cortex-53 processor. That might also help. Especially in getting the order correct.
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Fri Mar 09, 2018 11:09 am

Hi there,

thanks to both of you for your inputs. However, I did seem to have any luck there. Whatever I try there seem to be no change in behaviour.

The stackpointer looks fine to me. I've send them to the UART during startup and this is how they are:

Code: Select all

HYP stack pointer: 0x59DE8
SVC stack pointer: 0x99DE8
FIQ stack pointer: 0xA1DE8
IRQ stack pointer: 0xA9DE8
ABT stack pointer: 0xB1DE8
SYS stack pointer: 0xC1DE8
UND stack pointer: 0xC9DE8
When running the MMU-Init I'm also sending the stackpointer to the UART and it says it is 0xC1DE8 --> SYS mode...

Even when setting the domain access the Pi hangs when setting the MMU to enabled in the system control register. What is further weird is still, that setting the extended CPU control resgister with the mentioned code lets the Pi hang for a couple of seconds and than raising a RESET IRQ. I've tried to dump the registers to UART from the reset IRQ but I'm not sure whether those make any sense?

Code: Select all

CPU-ID: 0x0
r0......0x40
r1.....0x0
r2.....0xC1DD2
r3.....0x0
r4.....0xC34F
r5.....0x0
r6.....0x431BDE83
r7.....0x805C
r8.....0x8684
r9.....0x1C000
I'm not using any config.txt
The behaviour i the same, with and without bootloader :/
Well I'm somehow lost and have no further ideas where to start further analysing...
I'll go and reduce my testproject by everything except the MMU stuff.

Thx again...
Schnoogle

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Thu Mar 15, 2018 7:41 pm

What we haven’t discussed is the actual settings in the TLB and the memory attributes for the TTRB0. What actaul attributes did you use in the TLB and what memory attributes did you specify for TTRB0? Get these wrong or incompatible and the MMU will not start. I use 0x406A as memory attribute for TTRB0 and 0x90C0E for all the normal memory. What values did you use?
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Fri Mar 16, 2018 8:59 pm

Hi AlfredJingle,

well I tried several configurations I found in several baremetal source codes and forums. But unfortunately without any luck :/

The current setup uses your proposal.
When writing the TTBR0 register I'm using 0x406A as you do:

Code: Select all

        ldr		r0, =MMUTable
	ldr		r1, =0x406a
	orr		r0, r0, r1				
	mcr		p15, 0, r0, c2, c0, 0	// set TT base register 0 to MMUTable
The MMUTable entries use also your suggestion:

Code: Select all

.set 	SECTION, 0	// memory section/pages start at 0

.align 12
MMUTable:
.rept	4096						// repeat the 4096 entries
.word	SECTION + 0x90C0E			// table entry: section/page address + section/page attributes
.set	SECTION, SECTION + 0x100000 // next 1MB section/page
.endr
still no "luck".... the pi just hangs once it reaches this code to activate the MMU:

Code: Select all

mrc 	p15, 0, r0, c1, c0, 0
	orr 	r0, r0, #0x1				/* enable MMU */
	dsb
	mcr		p15, 0, r0, c1, c0, 0	/* <-- hangs here */
	isb

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Sat Mar 17, 2018 10:36 am

Hi schnoogle,

Aah, now we seem to be getting somwhere!

Your translation-table as such seems to be correct (as far as I understand C...) but while setting the TTBR0 you ORR MMUtable with 0x406a. But the 0x406a already contains the translation-table address. The first 4 in 0x406a means that the translation-table starts at 0x4000. If your translation-table (MMUtable as you call it) starts at another address, the 0x406a can't work. And in addition, if your translation-table address is not aligned to 0x1000 it won't work either.

So get rid of the ORR and MMUtable part in the first 4 lines of your code, just write 0x406a to TTBR0 and start your translation-table at 0x4000 and onwards. That should do it.
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Sat Mar 17, 2018 4:39 pm

Hi AlfredJingle,

thanks for your inputs.
My current MMUTable setup uses ".align 12" which put it at 0x1000 alignment. The current MMUTable was at 0x9000.
The "orr" with 0x406a" indeed does not make much sense here...

So I changed the setup to reflect your proposals.
The MMUTable/Translation table is now starting at 0x4000 an the first contents are:

Code: Select all

0x4000 = 0x90C0E
0x4004 = 0x190C0E
0x4008 = 0x290C0E
0x400C = 0x390C0E
0x4010 = 0x490C0E
0x4014 = 0x590C0E
0x4018 = 0x690C0E
0x401C = 0x790C0E
0x4020 = 0x890C0E
I've also checked the contents of the TTBR0 register which are:

Code: Select all

TTBR0: 0x406A
The memory domains are set to 0x55555555.

Unfortunately without success....:(

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Sun Mar 18, 2018 1:55 pm

Hi schnoogle,

I re-checked my system and I have exactly the same entries in my translation-table and the source for disabling and setting the MMU is essentially the same. See below (prolog, next and ldv32 are small macro's handling popping, pushing of, and loading of immediates into registers and the return from subroutines )

Things I would and you could look at next: you fill the whole of the translation table with the same settings (0x90C0E) but these settings are only valid for normal memory, not for the device-part of the memory map (0x3F000000 and above) and I have no info on the settings for memory above 0x40000000. I use 0x90C16 for that and the device-memory and that works. Check the ARMv8-ArchRefMan for an exact specification of the entries.

Another question is whether you have any other software running (you mention the UART-stuff) because if you have, than you not only have to disable the caches but you also might have to clean the caches (instruction-, main-TLB- and branche-prediction cache) My system starts fine without a data-cache clean, probably because I reset all essential variables during a restart. You write somewhere that an error should not lead to a mode-change but that is wrong. Errors in the software can lead to mode-change. It is for instance very easy to end up in HYP-mode with it's own register-bank. Maybe try to have as clean and small a system as possible. DWELCH has a great example how to enable the UART directly ( if you haven't already )

You could also try the play with the actual setting in the translation-table. For instance by not enabling the caches etc. Just to see whether you can get the MMU to start at all. And than work your way back to where the actual problem seems to be.

And than there is the whole issue of secure vs unsecure. The secure and unsecure modes have separate registers for the translation tabel! (allowing for different translation-tables for both modes) My system runs in secure mode and to be able to switch to secure-mode, I first have to set the translation-table related registers for the secure mode in MONITOR-mode = exception level 3. Before I managed to do that, I could only run in unsecure mode. Which runs fine but disallows access to some cpu-registers.

Happy bug-hunting!


Code: Select all

DISMMU:	prolog					@ dis MMU/caches, clean TLT/caches/branche prediction - inc databarier{
						@ functions for all cores
	@   ********* disable MMU and caches ***************************

		mrc p15, 0, r1, c1, c0, 0
		bic r1, r1, #1			@ disable MMU
		bic r1, r1, #4096		@ disable I cache
		bic r1, r1, #4			@ disable D cache
		mcr p15, 0, r1, c1, c0, 0
		isb
		
		mcr p15, 0, r0, c7, c5, 0	@ Invalidate Instruction Cache
		isb
		
		mcr p15, 0, r0, c7, c5, 6	@ BPIALL - invalidate all branch-prediction
		isb

		mov r0, #0
		mcr p15, 0, r0, c8, c7, 0	@ Invalidate entire Main TLB
		isb
next		@ checked }
 
MMUON:	prolog					@ { actual start MMU and caches and TTRB0 etc - for #core0 & #core1
		mov	r0, #0x0		@ use only TTBR0
		mcr	p15, 0, r0, c2, c0, 2	@ Write to TTBCR => simply put 0 in TTBCR
		isb

		ldv32 r1, 0x406A 		@ 0000 0000 0000 0000 0100 0000 0110 1010 shared inner, write back, alocate on write
						@ #ERROR: no allocate on write anymore!!!
		mcr	p15, 0, r1, c2, c0, 0	@ Write to TTBR0[31:0] - this is the non-secure versie!!
		isb

		ldv32 r0, 0xFFFFFFFF		@ set all access domains to 'manager' -> no access-checks
		mcr	p15, 0, r0, c3, c0, 0	@ Write to DACR
		isb

	@  ********************** start caches  ************************

		mrc p15, 0, r0, c1, c0, 0
		ldv32 r1, 0xfffffffd
		and r0, r0, r1			@ enable unaligned acces ( for string operations )
		orr r0, r0, #4096		@ instruction cache enabled
		orr r0, r0, #4			@ data cache enabled
		mcr p15, 0, r0, c1, c0, 0
		isb

	@  ****************** start MMU  *******************************

		mrc	p15, 0, r0, c1, c0, 0
		orr	r0, r0, #1		@ enables MMU
		mcr	p15, 0, r0, c1, c0, 0
		isb
next		@ checked }
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: RPi-3 (aarch32) - MMU activation hangs

Sat Mar 24, 2018 8:38 pm

Hi AlfredJingle,

thank you very much for your patient and your support in this. I guess I got it working now... :D :D :D
The trick seem to be that the MMU translation table is only excepted if located at 0x4000 I did not have any luck with any other location. And than your settings for the memory up to 0x3F00 0000 and after this was working now for me. Thanks so much... It feels a bit like a break through even that there is no documentation saying that the MMU does only work if setup in this excat way :(

Unfortunately since the MMU is active the access to the mailbox as described here: https://github.com/raspberrypi/firmware ... -mailboxes is no longer working - but this may be for another thread ?? --> got it, I needed too clear/invalidate caches, it's all working now ....

Nevertheless...thanks!
Schnoogle

deater
Posts: 27
Joined: Fri Mar 11, 2016 3:58 pm
Location: 45N

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Sun Mar 25, 2018 5:11 am

I have to say this was a really timely thread, I've spent the past few days trying to get MMU memory protection going on the Pi3 using my OS (http://www.deater.net/weave/vmwprod/vmwos/). I never would have figured out the proper page table settings without this thread.

On an added note, are caches working for you? MMU protection is working fine, but once I enable caches things run for a while but then fail with lots of weird memory errors. I try to flush the TLB, icache, and dcache before enabling things but it doesn't seem to be enough. Should I be limiting the L1 cache size to avoid aliasing? Do I need to flush the L2 cache too?

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Mon Mar 26, 2018 5:14 pm

Hi deater,

I've the caches active without any specific settings. Just settning the corresponding system control register flag and off we go. When accessing the mailbox interface to videocore I'm invalidating the L1 cache with code I've borrowed from the net and some ARM pages.

I'm doing heavy machine learing stuff that pokes the memory around a lot and it does seem to work quite well without any side effect. However, I'm not going to build an OS like you did. It is never intended to be so I guess there is lot of stuff an OS usually need I'm not implementing and therefore never see issues like you.

How could one limit or configure the cache size? I'm alos quite new to all those topics but if you share a bit more details on your "weird memory errors" we might be able to help.

BR Schnoogle

deater
Posts: 27
Joined: Fri Mar 11, 2016 3:58 pm
Location: 45N

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Thu Mar 29, 2018 12:17 pm

I figured out my problem. Some executable code was exiting, and new executable code was being loaded into the same location (via the dcache) but the icache had the old version, causing the weird memory errors. I added an icache flush to the loading code and now everything works with caches enabled.

The new problem I am having with the MMU being on is that reads from the GPU mailbox are now failing, I'm hoping that's just a pagetable flags setting I need to figure out.

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Thu Mar 29, 2018 4:17 pm

Hi deater,

well direct after MMU activation I had the same issue with the GPU mailbox.
I've done 2 things here.
I do a D-Cache invalidation direct before writing and after reading the mailbox data.
The address that is written to the mailbox containing the actual data is "ored" with 0xC0000000. This is the VC mapped "uncached" view of the address as I understood. The cache invalidation only ensures that the ARM CPU has written the real memory from cache before calling the GPU and update it's caches after data from GPU has arrived...

With this the MMU table entry flags could be kept as discussed in this thread - at least I keep them like this :)

BR Schnoogle

deater
Posts: 27
Joined: Fri Mar 11, 2016 3:58 pm
Location: 45N

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Fri Mar 30, 2018 11:39 pm

This works for me on Pi1 systems, but on my Pi3 I can't seem to get it to work. What instructions are you using to flush the cache on the Pi3?

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Sat Mar 31, 2018 11:39 am

Hi deater,

unfortunately there is not a single instruction or a hand-ful of instructions to flush the cache on RPi3..

This is the function I've grabbed from different resources and it works quite well, as far a I can tell it goes through the different cache levels and flushes the cache as required based on the cache lines...

Code: Select all

/****************************************************
 * cache.s
 *
 * cache management functions
 ****************************************************/
 .global invalidate_dcache

/*
 *************************************************************************
 *
 * invalidate_dcache - invalidate the entire d-cache by set/way
 *
 * Note: for Cortex-A53, there is no cp instruction for invalidating
 * the whole D-cache. Need to invalidate each line.
 *
 *************************************************************************
 */
invalidate_dcache:
	push	{r0 - r12}
	mrc	p15, 1, r0, c0, c0, 1		/* read CLIDR */
	ands	r3, r0, #0x7000000
	mov	r3, r3, lsr #23			/* cache level value (naturally aligned) */
	beq	finished
	mov	r10, #0				/* start with level 0 */
loop1:
	add	r2, r10, r10, lsr #1		/* work out 3xcachelevel */
	mov	r1, r0, lsr r2			/* bottom 3 bits are the Cache type for this level */
	and	r1, r1, #7			/* get those 3 bits alone */
	cmp	r1, #2
	blt	skip				/* no cache or only instruction cache at this level */
	mcr	p15, 2, r10, c0, c0, 0		/* write the Cache Size selection register */
	isb					/* isb to sync the change to the CacheSizeID reg */
	mrc	p15, 1, r1, c0, c0, 0		/* reads current Cache Size ID register */
	and	r2, r1, #7			/* extract the line length field */
	add	r2, r2, #4			/* add 4 for the line length offset (log2 16 bytes) */
	ldr	r4, =0x3ff
	ands	r4, r4, r1, lsr #3		/* r4 is the max number on the way size (right aligned) */
	clz	r5, r4				/* r5 is the bit position of the way size increment */
	ldr	r7, =0x7fff
	ands	r7, r7, r1, lsr #13		/* r7 is the max number of the index size (right aligned) */
loop2:
	mov	r9, r4				/* r9 working copy of the max way size (right aligned) */
loop3:
	orr	r11, r10, r9, lsl r5		/* factor in the way number and cache number into r11 */
	orr	r11, r11, r7, lsl r2		/* factor in the index number */
	mcr	p15, 0, r11, c7, c6, 2		/* invalidate by set/way */
	subs	r9, r9, #1			/* decrement the way number */
	bge	loop3
	subs	r7, r7, #1			/* decrement the index */
	bge	loop2
skip:
	add	r10, r10, #2			/* increment the cache number */
	cmp	r3, r10
	bgt	loop1

finished:
	mov	r10, #0				/* swith back to cache level 0 */
	mcr	p15, 2, r10, c0, c0, 0		/* select current cache level in cssr */
	dsb
	isb

	pop {r0-r12}
	bx	lr
Hope this helps solving your problem...

BR Schnoogle

AlfredJingle
Posts: 69
Joined: Thu Mar 03, 2016 10:43 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Fri Apr 06, 2018 6:15 pm

Hi,

Yeah, I use that routine for cleaning of the datacaches as well. I adapted it a bit so that it takes as input which of the three principle cache-actions it has to perform:

Namely either:

Code: Select all

mcr p15, 0, r11, c7, c6, 2 @ = DCISW - datacache invalidate by set/way
or		
mcr p15, 0, r11, c7, c10, 2 @ = DCCSW - datacache clean by set/way
or
mcr p15, 0, r11, c7, c14, 2 @ = DCCISW - datacache clean and invalidate by set/way
If you only invalidate a datacache it can happen that data in actual memory is not up to date. A clean of the datacache ensures that all data contained in the cache is also visible in memory. Before I understood and implemented that, I had spurious bugs which happened now and than, and which had no obvious reason. Drove me almost crazy for a while!
going from a 6502 on an Oric-1 to an ARMv8 is quite a big step...

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Mon Apr 09, 2018 4:33 pm

Hey AlfredJingle,

thanks for the hint, I will update my function accordingly :)

BR Schnoogle

deater
Posts: 27
Joined: Fri Mar 11, 2016 3:58 pm
Location: 45N

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Wed Apr 11, 2018 6:19 pm

sorry for the delay in replying. Using dcache clearing code seems to work fine for getting the GPU framebuffer to work.

I was trying to use the same code for accessing the mailbox to get the current temperature, and no matter how I clear the caches I cannot get the temperature tags interface to work on a Pi3 with the MMU enabled. (The code works properly and returns the temperature if I don't enable the MMU, and it works fine on a pi1+MMU). Though maybe I should ask this question in it's own topic instead of continuing on here.

Schnoogle
Posts: 41
Joined: Sun Feb 11, 2018 4:47 pm

Re: [solved] RPi-3 (aarch32) - MMU activation hangs

Thu Apr 12, 2018 1:04 pm

Hi deater,

well - it might be really the best - also for others to follow - when you open a separate thread. However, I'm using the mailbox interface for getting CPU speed (ARM and Core) as well as getting the ARM and GPU memory split with MMU active on RPi3 without any issues so far.
However, let's discuss in a new thread ;)

BR Schnoogle

Return to “Bare metal, Assembly language”

Who is online

Users browsing this forum: No registered users and 6 guests