Error in GPIO interrupt handling

[ceggers]

Hardware: RPi 3B

I have problems with consecutive interrupts (edge triggered) on two different GPIO pins. In my application, the second interrupt often appears about 5 µs after the first interrupt. When the 2nd IRQ (e.g GPIO6) arrives exactly when the 1st IRQ (e.g. GPIO5) is handled, the 2nd IRQ will sometimes not be raised (depending on the timing). If the 2nd IRQ appears earlier or later than 5 µs, everything is fine.

After some days of investigation I'm quite sure, that the GPEDS register of the GPIO controller is not implemented correctly (controller bug). Everytime the Linux driver (pinctrl-bcm2835.c) "acks" an interrupt by writing to GPEDS0, a simultaneous IRQ on another GPIO pin will be lost. Although the problematic "window" may be only one clock cycle long, I can easily reproduce this problem within a few minutes.

For demonstration purposes, I've written a small test case (only the RPi 3B and a piece of wire is required). In this test case, I try to activate simultaneous access (setting one event and reset another one) of the GPEDS0 register. Setting is done via external GPIO loop (GPIO output --> IRQ interrupt). Resetting is done via a C application.

Test setup:
1. Make sure that GPIO2 and GPIO3 are not used (probably disable I2C in /boot/firmware/config.txt and reboot)
2. Connect GPIO2 and GPIO3 with an external piece of wire
3. Configure GPIO2 and GPIO3 via sysfs

Code: Select all

cd /sys/class/gpio/
echo 2 > export
echo out > gpio2/direction
echo 3 > export
echo falling > gpio3/edge

4. Check number of interrupts

Code: Select all

grep gpiolib /proc/interrupts
169:          0          0          0          0  pinctrl-bcm2835   3 Edge      gpiolib

5. Generate 100 interrupts (you can omit the sleep statement or use other values)

Code: Select all

I=0 && while [ $I -lt 100 ]; do echo 1 > gpio2/value && echo 0 > gpio2/value && I=$(($I+1)) && sleep 0.1; done

6. Check number of interrupts

Code: Select all

grep gpiolib /proc/interrupts
169:        100          0          0          0  pinctrl-bcm2835   3 Edge      gpiolib

7. Run the following C program on another console

Code: Select all

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>


int main(void)
{
	static const uint32_t gpioGpeds0 = 0x40 / sizeof(uint32_t);

	int fd = open("/dev/gpiomem", O_RDWR | O_SYNC | O_CLOEXEC);
	if (fd < 0)
	{
		fprintf(stderr, "Cannot open /dev/gpiomem: %s\n", strerror(errno));
		exit(EXIT_FAILURE);
	}

	uint32_t *gpio = (uint32_t *)mmap(NULL, 4*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x00200000);
	if (gpio == MAP_FAILED)
	{
		fprintf(stderr, "mmap() failed: %s\n", strerror(errno));
		exit(EXIT_FAILURE);
	}

	while (1)
	{
		/* Reset Event Detect Status for bit 1 (you can try every bit, even writing 0 will work)*/
		*(gpio + gpioGpeds0) = (1 << 0);
	}

}

8. Generate another 100 interrupt while the C program is running

Code: Select all

I=0 && while [ $I -lt 100 ]; do echo 1 > gpio2/value && echo 0 > gpio2/value && I=$(($I+1)) && sleep 0.1; done

9. Check number of interrupts

Code: Select all

grep gpiolib /proc/interrupts
169:        148          0          0          0  pinctrl-bcm2835   3 Edge      gpiolib

The number of "lost" interrupts may vary from run to run. For me it looks like the GPIO edge detection cannot assert a bit in GPEDS when the CPU performs a write access to this register at the same time. The value which is written by the CPU doesn't care, even writing 0 seems raise the error.

[joan]

Depending on what you are trying to achieve you may get more useful results by using pigpio. By default it doesn't use Linux interrupts to monitor the GPIO state so will bypass any limitations of the hardware event registers.

[piras77]

After some days of investigation I'm quite sure, that the GPEDS register of the GPIO controller is not implemented correctly (controller bug).

In my software I directly access the peripherals from userland. And in my experience, the event detect status register works fine.

In my application, the second interrupt often appears about 5 µs after the first interrupt. When the 2nd IRQ (e.g GPIO6) arrives exactly when the 1st IRQ (e.g. GPIO5) is handled, the 2nd IRQ will sometimes not be raised (depending on the timing). If the 2nd IRQ appears earlier or later than 5 µs, everything is fine.

Well, this is related to a Raspbian driver that handles these interrupts. In general, interrupt processing may take some time. If another interrupt arrives before the first one is processed, it may get lost.

Still, the bit for the other GPIO pin should get raised if a related event occurs while the ISR is in progress. So the ISR *should* check at the end (before it finishes) such condition and act accordingly. If it does, and actually what it does, is beyond my knowledge. In the past I'v been under the impression, that there may be driver/kernel activities that reset the event detect status register (viewtopic.php?f=66&t=192908&p=1209100#p1209100) though I couldn't make a reliable test case for that.

[ceggers]

Depending on what you are trying to achieve you may get more useful results by using pigpio. By default it doesn't use Linux interrupts to monitor the GPIO state so will bypass any limitations of the hardware event registers.

pigpio looks quite interesting, but it seems that it uses "samling" instead of interrupts (didn't find this on the homepage, would have to investigate the source code). For my application I don't want to waste CPU time for this ...

[ceggers]

After some days of investigation I'm quite sure, that the GPEDS register of the GPIO controller is not implemented correctly (controller bug).

In my software I directly access the peripherals from userland. And in my experience, the event detect status register works fine.

What I've found is quite a corner case, which will not appear on many use cases. In practice you would need two different IRQs lines on the same GPIO bank. If the 2nd GPIO IRQ line is toggled exactly when the 1st line is reset (write access to GPEDS), the 2nd IRQ will be lost.

In my application, the second interrupt often appears about 5 µs after the first interrupt. When the 2nd IRQ (e.g GPIO6) arrives exactly when the 1st IRQ (e.g. GPIO5) is handled, the 2nd IRQ will sometimes not be raised (depending on the timing). If the 2nd IRQ appears earlier or later than 5 µs, everything is fine.

Well, this is related to a Raspbian driver that handles these interrupts. In general, interrupt processing may take some time. If another interrupt arrives before the first one is processed, it may get lost.

I know that a consecutive IRQ on the SAME line can not be recognized if the previous IRQ is still pending. But I'm talking about ANOTHER irq.

Still, the bit for the other GPIO pin should get raised if a related event occurs while the ISR is in progress. So the ISR *should* check at the end (before it finishes) such condition and act accordingly. If it does, and actually what it does, is beyond my knowledge. In the past I'v been under the impression, that there may be driver/kernel activities that reset the event detect status register (viewtopic.php?f=66&t=192908&p=1209100#p1209100) though I couldn't make a reliable test case for that.

I've read the sourcecode of the linux driver. I think that the driver only resets the bit which is currently handled. Other bits are not reset here.

[piras77]

I honestly can't see any proof of a faulty GPEDS0 peripheral.

I can't either follow your argument. You know there is only one interrupt "line" for the whole GPIO bank 0?!

I can't even reproduce your experiment in your first post: I get a 100 increment whether your program runs or not. Raspbian image 2017-09-07. However, I get (unsurprisingly) no increment at all if I change the reset bit to 0x8 (for pin 3) and declare the pointer volatile. Anyway, neither is proof for a faulty GPEDS0 peripheral.

If you want to provide a test case for a faulty GPEDS0 peripheral, I would advise to poll the GPEDS0 register and present the results. Block interrupts while your test case is in progress to prevent kernel intervention.

However, maybe I didn't understand your point. If so, sorry for that.

[ceggers]

I can't even reproduce your experiment in your first post: I get a 100 increment whether your program runs or not. Raspbian image 2017-09-07.

- I can confirm that my bug seems not to appear on Raspian image 2017-09-07.
- My problem is perfectly reproducible on Debian (https://jrwr.io/rpi/)

Thank you very much for spending time on investigating on my problem. Now I'll try to figure the differences between my Raspian and Debian system (the Debian system uses nearly the same kernel as the Raspian system...).

[ceggers]

- When I BUILD the test executable under Raspbian and RUN under Raspbian, NO interrupts are lost.
- When I BUILD the test executable under Debian and RUN under Raspbian, about 10 interrupts are lost.

...