tom66
Posts: 18
Joined: Tue Jan 10, 2012 12:14 am

AXI performance monitors - how to enable

Sat May 16, 2020 1:00 pm

Hi all,

I want to enable the AXI performance monitors to debug a potential performance issue we're having with the GPU and CSI bus.

I added:

Code: Select all

dtparam=axiperf=on
to the /boot/cmdline.txt, and can see this cmdline in dmesg. However there is no directory available, as promised:

Code: Select all

[raspberrypi:~]% sudo cat /sys/kernel/debug/raspberrypi_axi_monitor
cat: /sys/kernel/debug/raspberrypi_axi_monitor: No such file or directory
What's missing?

We are using a CM3 with kernel:

Code: Select all

Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux
This is running latest Raspbian (Buster, Feb 2020).

I noticed that there doesn't seem to be a reference to axiperf in the dts for the CM3 ... could this be a problem? (Edit: Nevermind, I think it's in a common file. I decompiled our DTB and there's references to the relevant registers there, so the mystery deepens.)

Many thanks
Last edited by tom66 on Sat May 16, 2020 1:06 pm, edited 1 time in total.

User avatar
dickon
Posts: 1874
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: AXI performance monitors - how to enable

Sat May 16, 2020 1:04 pm

Code: Select all

modprobe raspberrypi_axi_monitor
any good?
As it is apparently board policy to disallow any criticism of anything, as it appears to criticise something is to criticise all the users of that something, I will no longer be commenting in threads which are not directly relevant to my uses of the Pi.

tom66
Posts: 18
Joined: Tue Jan 10, 2012 12:14 am

Re: AXI performance monitors - how to enable

Sat May 16, 2020 1:12 pm

It exits with no error printed but

Code: Select all

sudo cat /sys/kernel/debug/raspberrypi_axi_monitor

still does not bring anything up (same not found error).

The first time I ran it, I got some dmesg warning about a page fault... not sure if it is just a coincidence:

Code: Select all

[  789.138028] Unhandled fault: external abort on non-linefetch (0x1008) at 0x76fa7000
[  789.145694] pgd = 2d6e2cea
[  789.148394] [76fa7000] *pgd=0fe00835, *pte=3f4e6783, *ppte=3f4e6e33
[  798.662535] Unhandled fault: external abort on non-linefetch (0x1008) at 0x76f32000
[  798.670226] pgd = 2d6e2cea
[  798.672936] [76f32000] *pgd=350fd835, *pte=3f4e6783, *ppte=3f4e6e33
I think it probably is given the address for the AXIPERF module is listed as 0x7e009800 here: https://paulwratt.github.io/rpi-interna ... PERF0.html

trejan
Posts: 3062
Joined: Tue Jul 02, 2019 2:28 pm

Re: AXI performance monitors - how to enable

Sat May 16, 2020 7:09 pm

tom66 wrote:
Sat May 16, 2020 1:00 pm

Code: Select all

dtparam=axiperf=on
to the /boot/cmdline.txt
Wrong file. You need to add it to /boot/config.txt

The raspberrypi_axi_monitor module is loaded automatically.

User avatar
dickon
Posts: 1874
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: AXI performance monitors - how to enable

Sat May 16, 2020 9:19 pm

Ooph. How did I miss *that*..?
As it is apparently board policy to disallow any criticism of anything, as it appears to criticise something is to criticise all the users of that something, I will no longer be commenting in threads which are not directly relevant to my uses of the Pi.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27708
Joined: Sat Jul 30, 2011 7:41 pm

Re: AXI performance monitors - how to enable

Sat May 16, 2020 9:40 pm

Good grief, someone actually using the axi performance monitor! Have fun. If it works, I wrote it, if it doesn't I have no idea who wrote it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

tom66
Posts: 18
Joined: Tue Jan 10, 2012 12:14 am

Re: AXI performance monitors - how to enable

Sun May 17, 2020 10:53 am

Thanks, that sorted that. I get the performance counters now in a directory listing under /sys/kernel/debug/raspberrypi_axi_monitor

I have a dangerous level of misunderstanding of AXI after having played with it on an FPGA for the past 6 months, so hopefully I can make sense of a problem I'm getting. Will post back if I need any more help.

cleverca22
Posts: 2345
Joined: Sat Aug 18, 2012 2:33 pm

Re: AXI performance monitors - how to enable

Mon May 18, 2020 4:51 am

could you also post back if you confirm its all working right and what you learned about how it actually works? ive also been curious about enabling it and poking around, and sharing what we know could speed things up

tom66
Posts: 18
Joined: Tue Jan 10, 2012 12:14 am

Re: AXI performance monitors - how to enable

Thu May 21, 2020 4:11 pm

The performance counters work, there appear to be three channels roughly inline with AXI. You can look up the interface on Wikipedia, there is a very comprehensive set of documentation there. In addition it is a freely available standard from ARM.

Regarding AXI, it essentially is a way for devices to access external peripherals and main memory space in a uniform manner that can be routed across a processor and occasionally along short board lengths between e.g. FPGAs (though it's not optimised for this!) I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface. It essentially exposes this into a "safed" protocol with read/write handshaking, write acknowledgement, and bursting (which improves performance.) It can be a very complex protocol when you start diving into some of the advanced behaviours but at its heart is relatively simple. I have implemented an AXI-Stream and AXI-Lite controller on an FPGA to learn how this all works and you can do GB/s transfer rates pretty easily.

Anyway, on the Pi AXI performance counters, you have read (R), write (W) and address (A). I am guessing the address-read and address-write channels are combined into a single statistic and write-response is not logged but I have not looked deeply enough into this to confirm, someone from the Foundation can probably answer if needed. The counters measure bus totals in the window they are active in, by default 100ms which is round-robin'd around the channels. When I do more work on the GPU, I can see the data reads and writes. Similarly for the camera peripherals.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27708
Joined: Sat Jul 30, 2011 7:41 pm

Re: AXI performance monitors - how to enable

Thu May 21, 2020 6:00 pm

tom66 wrote:
Thu May 21, 2020 4:11 pm
The performance counters work, there appear to be three channels roughly inline with AXI. You can look up the interface on Wikipedia, there is a very comprehensive set of documentation there. In addition it is a freely available standard from ARM.

Regarding AXI, it essentially is a way for devices to access external peripherals and main memory space in a uniform manner that can be routed across a processor and occasionally along short board lengths between e.g. FPGAs (though it's not optimised for this!) I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface. It essentially exposes this into a "safed" protocol with read/write handshaking, write acknowledgement, and bursting (which improves performance.) It can be a very complex protocol when you start diving into some of the advanced behaviours but at its heart is relatively simple. I have implemented an AXI-Stream and AXI-Lite controller on an FPGA to learn how this all works and you can do GB/s transfer rates pretty easily.

Anyway, on the Pi AXI performance counters, you have read (R), write (W) and address (A). I am guessing the address-read and address-write channels are combined into a single statistic and write-response is not logged but I have not looked deeply enough into this to confirm, someone from the Foundation can probably answer if needed. The counters measure bus totals in the window they are active in, by default 100ms which is round-robin'd around the channels. When I do more work on the GPU, I can see the data reads and writes. Similarly for the camera peripherals.
I wrote that driver quite some time ago, and almost immediately forgot how it worked. WIll need to check the datasheets etc, will try and remember to do that tomorrow.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

cleverca22
Posts: 2345
Joined: Sat Aug 18, 2012 2:33 pm

Re: AXI performance monitors - how to enable

Fri May 22, 2020 10:27 am

tom66 wrote:
Thu May 21, 2020 4:11 pm
I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface.
my rough understanding of the AXI layout, is that the arm cores run thru the ARM_TRANSLATE unit, which deals with mapping arm-ram to vpu 0xc000_0000, and arm-mmio to 0x7e00_0000

the main responders in that vpu domain, are then the ram, mmio hw (many ports?) and the L2 cache (which will reflect the request back out on a different range upon miss)

and the dma/vpu/hvs (and others) can also request data within that domain

for the rpi1, the arm had no cache of its own, and was commonly configured(via ARM_TRANSLATE) to leech off the vpu L2 cache, to get some minor performance improvements


the rpi4 redid everything, and ive not gotten a good grasp of how it works yet

Return to “Advanced users”