dylan1280
Posts: 2
Joined: Mon Apr 23, 2018 10:20 pm

Interrupt differences between 32-bit and 64-bit kernels

Tue Apr 24, 2018 2:55 pm

I have a Raspberry Pi 3 Model B V1.2 that I am evaluating for real time processing. When testing on other ARM and x86 platforms I have found it helpful to set the smp_affinity of IRQs to core 0 while running the real time process on the remaining cores. My process requires the 64-bit architecture. I have read that it is not possible to change the affinity of individual IRQs.

Code: Select all

root@localhost:~# echo 1 > /proc/irq/41/smp_affinity
bash: echo: write error: Input/output error

I have noticed that on the 32-bit kernel interrupts mostly occur on core 0 and on the 64-bit kernel interrupts are more evenly distributed among all cores. Is there is any known explanation for this difference? Outputs from dmesg and /proc/interrupts below:

64-bit Kernel:

Code: Select all

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.34-v8+ (root@dylansnuc) (gcc version 5.4.1 20161213 (Linaro GCC 5.4-2017.01)) #2 SMP PREEMPT Wed Apr 18 18:06:57 EDT 2018
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] Machine model: Raspberry Pi 3 Model B
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 8 MiB at 0x000000003a400000
[    0.000000] On node 0 totalpages: 242688
[    0.000000]   DMA zone: 3792 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 242688 pages, LIFO batch:31
[    0.000000] random: fast init done
[    0.000000] percpu: Embedded 22 pages/cpu @ffffffc03b379000 s51712 r8192 d30208 u90112
[    0.000000] pcpu-alloc: s51712 r8192 d30208 u90112 alloc=22*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 238896
[    0.000000] Kernel command line: ip=dhcp root=/dev/mmcblk0p2 rw noirqbalance isolcpus=1-3 rcu_nocbs=1-3 nohz_full=1-3 irqaffinity=0 initrd=0x02100000,0x54c1eb

Code: Select all

           CPU0       CPU1       CPU2       CPU3       
  2:   58143081       2840       2729       2675  bcm2836-timer   1 Edge      arch_timer
  6:       7661       6643       7137       7070  ARMCTRL-level   1 Edge      3f00b880.mailbox
  7:          1          0          0          0  ARMCTRL-level   2 Edge      VCHIQ doorbell
 15:    7959396    7647122    7956896    7666630  ARMCTRL-level  64 Edge      dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1
 41:  121832918  121367565  121771070  121552960  ARMCTRL-level  41 Edge      dwc_otg_sim-fiq
 48:          0          0          0          0  ARMCTRL-level  48 Edge      bcm2708_fb dma
 50:          0          0          0          0  ARMCTRL-level  50 Edge      DMA IRQ
 52:          0          0          0          0  ARMCTRL-level  52 Edge      DMA IRQ
 53:          0          0          0          0  ARMCTRL-level  53 Edge      DMA IRQ
 56:        541        601        538        587  ARMCTRL-level  56 Edge      DMA IRQ
 61:          0          0          0          0  ARMCTRL-level  61 Edge      bcm2835-auxirq
 71:        154        134        140        173  ARMCTRL-level  88 Edge      mmc0
 77:        212        247        213        219  ARMCTRL-level  94 Edge      mmc1
IPI0:    485616       1822       1747       1710       Rescheduling interrupts
IPI1:         6       1295       1274       1252       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:    548890      85226      85043      85336       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

32-bit Kernel:

Code: Select all

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.34-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1110 SMP Mon Apr 16 15:18:51 BST 2018
[    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B Rev 1.2
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] cma: Reserved 8 MiB at 0x3ac00000
[    0.000000] On node 0 totalpages: 242688
[    0.000000] free_area_init_node: node 0, pgdat 80c84e40, node_mem_map ba3a1000
[    0.000000]   Normal zone: 2133 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 242688 pages, LIFO batch:31
[    0.000000] random: fast init done
[    0.000000] percpu: Embedded 17 pages/cpu @ba34a000 s38720 r8192 d22720 u69632
[    0.000000] pcpu-alloc: s38720 r8192 d22720 u69632 alloc=17*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 240555
[    0.000000] Kernel command line: 8250.nr_uarts=0 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1200 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles nohz_full=1-3 rcu_nocbs=1-3 noirqbalance isolcpus=1-3

Code: Select all

           CPU0       CPU1       CPU2       CPU3       
 16:          0          0          0          0  bcm2836-timer   0 Edge      arch_timer
 17:      48398          8          8          8  bcm2836-timer   1 Edge      arch_timer
 21:          0          0          0          0  bcm2836-pmu   9 Edge      arm-pmu
 23:        241          0          0          0  ARMCTRL-level   1 Edge      3f00b880.mailbox
 24:          7          0          0          0  ARMCTRL-level   2 Edge      VCHIQ doorbell
 46:          0          0          0          0  ARMCTRL-level  48 Edge      bcm2708_fb dma
 48:          0          0          0          0  ARMCTRL-level  50 Edge      DMA IRQ
 50:          0          0          0          0  ARMCTRL-level  52 Edge      DMA IRQ
 51:        755          0          0          0  ARMCTRL-level  53 Edge      DMA IRQ
 54:      10496          0          0          0  ARMCTRL-level  56 Edge      DMA IRQ
 59:          0          0          0          0  ARMCTRL-level  61 Edge      bcm2835-auxirq
 62:     260789          0          0          0  ARMCTRL-level  64 Edge      dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1
 86:       1129          0          0          0  ARMCTRL-level  88 Edge      mmc0
 87:       5141          0          0          0  ARMCTRL-level  89 Edge      uart-pl011
 92:      67226          0          0          0  ARMCTRL-level  94 Edge      mmc1
FIQ:              usb_fiq
IPI0:          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0  Timer broadcast interrupts
IPI2:         25        136        136        134  Rescheduling interrupts
IPI3:          0          6          5          4  Function call interrupts
IPI4:          0          0          0          0  CPU stop interrupts
IPI5:        541          0          0          0  IRQ work interrupts
IPI6:          0          0          0          0  completion interrupts
Err:          0

swahren
Posts: 140
Joined: Mon Sep 19, 2016 5:24 pm
Location: Germany

Re: Interrupt differences between 32-bit and 64-bit kernels

Sun Apr 29, 2018 9:38 am

As you noticed the smp_affinity isn't implemented yet. The difference in the behaviour comes from this patch:

https://github.com/raspberrypi/linux/co ... a99570d782

dylan1280
Posts: 2
Joined: Mon Apr 23, 2018 10:20 pm

Re: Interrupt differences between 32-bit and 64-bit kernels

Tue May 01, 2018 4:33 pm

Thank you! This patch was exactly what I was looking for.

phoyer
Posts: 1
Joined: Thu Jan 30, 2020 10:23 am
Location: Italy

Re: Interrupt differences between 32-bit and 64-bit kernels

Thu Jan 30, 2020 10:28 am

swahren wrote:
Sun Apr 29, 2018 9:38 am
As you noticed the smp_affinity isn't implemented yet.
Is the smp_affinity ever going to be implemented? It is an important feature to achieve performance in case of IRQ-hungry peripherals...

swahren
Posts: 140
Joined: Mon Sep 19, 2016 5:24 pm
Location: Germany

Re: Interrupt differences between 32-bit and 64-bit kernels

Fri Jan 31, 2020 8:05 pm

phoyer wrote:
Thu Jan 30, 2020 10:28 am
swahren wrote:
Sun Apr 29, 2018 9:38 am
As you noticed the smp_affinity isn't implemented yet.
Is the smp_affinity ever going to be implemented? It is an important feature to achieve performance in case of IRQ-hungry peripherals...
Nobody asked for this before as a necessary feature. There are only few kernel developer working on upstream and all of them concentrate on RPi 4.
Which generation of RPi you had in mind?

Return to “Linux Kernel”