9pi
Posts: 46
Joined: Sat Aug 11, 2012 6:14 pm

PCIe outbound address restrictions

Fri Sep 13, 2019 8:40 pm

Is it possible to map the PCIe outbound window (base address for requests from CPU to PCIe) to an address below 4GiB, ie to a 32-bit address?

I know that linux maps the window at 0x600000000, which requires LPAE on arm32. I'm working on PCIe support for Plan 9 on arm32, which doesn't use LPAE. I've tried mapping the window at 0x80000000 for example (on a 2GiB pi4), or at 0xFBF00000 (sacrificing 1MiB of ram on a 4GiB pi4), but neither seems to work (usb3 controller registers in the PCIe window just read as zeroes).

Even on an arm64 variant of Plan 9, it appears that mapping the PCIe outbound window at 0x600000000 works but 0x80000000 does not.

Is there something in the Raspberry Pi memory architecture which restricts the configured PCIe window addresses to be >4GiB?

elahav
Posts: 21
Joined: Fri Jan 18, 2019 11:08 am

Re: PCIe outbound address restrictions

Wed Sep 18, 2019 1:52 am

Don't you have a conflict with RAM if you try to use the lower addresses? There is no data sheet for the RPi 4 (at least nothing I could find), but it seems that all addresses from 0 to 0xfc000000 are taken for RAM.

9pi
Posts: 46
Joined: Sat Aug 11, 2012 6:14 pm

Re: PCIe outbound address restrictions

Wed Sep 18, 2019 8:46 am

elahav wrote:
Wed Sep 18, 2019 1:52 am
Don't you have a conflict with RAM if you try to use the lower addresses?
Yes, that's what I meant by the need to "sacrifice" a megabyte of RAM if mapping pcie outbound addresses at 0xFBF00000 on a 4GiB pi4. On a 2GiB pi4 there should be not conflict. A comment in the linux driver (pcie-brcmstb.c) says:

Code: Select all

* ... If a portion of the
* viewport does not represent system memory -- e.g. 3GB of
* memory requires a 4GB viewport -- we can map the outbound
* memory in or after 3GB an even though the viewport will
* overlap the outbound memory the controller will know
* to send outbound memory downstream and everything else
* upstream
.
Here "viewport" means the inbound address space which the pcie uses to access RAM. The comment seems to imply that it's OK for outbound and inbound spaces to overlap.

But maybe the SCB bus in the BCM2711 adds an extra complication?

elahav
Posts: 21
Joined: Fri Jan 18, 2019 11:08 am

Re: PCIe outbound address restrictions

Wed Sep 18, 2019 10:49 am

For what it's worth, I looked at the first 4KB of memory when mapping outbound PCI addresses at 0x6_0000_0000, 0x8000_0000 and 0x2_0000_0000 on a pure 64-bit OS (QNX), and only the first of those seems legit (though I am by no means a PCI expert).

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2160
Joined: Thu Jul 11, 2013 2:37 pm

Re: PCIe outbound address restrictions

Wed Sep 18, 2019 10:51 am

The outbound window is fixed in the bus address range 0x6_0000_0000 to 0x7_ffff_ffff. There can be up to 3 subdivisions of this window as there's 3 sets of window registers.

Also, the smallest window size supported by the outbound window registers is 1MB.
Rockets are loud.
https://astro-pi.org

9pi
Posts: 46
Joined: Sat Aug 11, 2012 6:14 pm

Re: PCIe outbound address restrictions

Wed Sep 18, 2019 11:03 am

jdb wrote:
Wed Sep 18, 2019 10:51 am
The outbound window is fixed in the bus address range 0x6_0000_0000 to 0x7_ffff_ffff.
Fixed by the hardware? That's the conclusion I was approaching, after doing some linux experiments today, by changing the ranges in bcm2838.dtsi and finding which ones worked.

Thanks for the confirmation.

Return to “Bare metal, Assembly language”