hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Memory barriers for GPIO access

Thu May 23, 2019 10:38 am

The Broadcom BCM 2835 Peripheral PDF talks of read and write memory barriers which should be executed before the first read and after the last write (page 7) -

https://www.raspberrypi.org/app/uploads ... herals.pdf

This is because one isn't actually updating things at the instant a register is written or read but passing commands along a bus to get things updated or read. Don't use the barriers and things can go screwy.

But I'm not exactly sure when they need to be used or how they are specified, and most of the GPIO interfacing libraries don't seem to use them nor even mention them.

Does that mean they aren't actually needed, don't need to be explicitly specified, or are those libraries simply less safe and less reliable than they should be ?

Are memory barriers only required for the original BCM 2835 or also for later variants ?

I thought the GPIO peripheral block was the same for all Pi variants and those barriers should therefore be there.

Can anyone shed any light on the issue ?

User avatar
procount
Posts: 1616
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Memory barriers for GPIO access

Thu May 23, 2019 10:54 am

Have a read of this thread -->viewtopic.php?f=72&t=234420&p=1447526&h ... s#p1447526

Just pure speculation on my behalf, but my take on it is that unless you are writing baremetal code, or kernel drivers, you can safely ignore it, as normal user code/libraries are so far abstracted from this issue that it is all taken care of under the hood by the linux drivers themselves. But if you are writing baremetal/kernel, then you will need to take care of it.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Thu May 23, 2019 11:33 am

The libraries I was thinking of are those like RPi.GPIO and Zero.GPIO called by Python. Those are C extensions, but it's pretty much bare-metal as access to GPIO is done as a direct read or write of memory of the type -

*(gpio+offset) = value;
return *(gpio+offset);

For example lines 260 and 269 here for RPi.GPIO ...

https://sourceforge.net/p/raspberry-gpi ... e/c_gpio.c

Not a memory barrier in sight.

It seems to me that bare-metal might actually be the better case in arguing for less need of using memory barriers, because execution would be purely linear and non-interruptible in many cases. On a real-time, multi-core, multi-threaded, interruptible system, one cannot guarantee what sequence is happening, exactly what gets executed, in what order or how GPIO access gets interleaved.

I was also thinking there's an argument they have to be used in all cases or there's a risk the GPU might jump in and access the PMIC's I2C channel or similar and screw what the userland program expects.

In the worst case, I guess, if some non-barriered operation screwed with ongoing access to the SD Card controller, that could lead to all sorts of problems, corruption and worse.

dl324
Posts: 109
Joined: Mon May 06, 2019 7:33 pm
Location: Pacific Northwest, USA

Re: Memory barriers for GPIO access

Thu May 23, 2019 2:58 pm

hippy wrote:
Thu May 23, 2019 10:38 am
The Broadcom BCM 2835 Peripheral PDF talks of read and write memory barriers which should be executed before the first read and after the last write (page 7) -

https://www.raspberrypi.org/app/uploads ... herals.pdf

This is because one isn't actually updating things at the instant a register is written or read but passing commands along a bus to get things updated or read. Don't use the barriers and things can go screwy.
That only applies for accesses to multiple peripherals (GPIO, SPI, PWM, etc). You shouldn't have any problems if you're only accessing GPIO.

When I write something to a GPIO register, it's updated when the write completes.

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Thu May 23, 2019 3:46 pm

dl324 wrote:
Thu May 23, 2019 2:58 pm
That only applies for accesses to multiple peripherals (GPIO, SPI, PWM, etc). You shouldn't have any problems if you're only accessing GPIO.
But what if I am accessing GPIO and something else then interrupts and accesses some other peripheral, or another parallel core or GPU accesses a different peripheral, and I don't have a barrier protecting things ?

Code: Select all

My core                         | Some other core
                                |
myStatus = *MY_PERIPHERAL_PTR;  |
:                               | itsSatus = *ITS_PERIPHERAL_PTR;
return myStatus;                | :
                                | return itsStatus;
That seems to be exactly what the ARM Peripheral PDF warns against, says can cause problems. And I can create exactly that just by running two Python programs. Of course the chance of things coinciding is slim, but not impossible. Likely possible enough to show a few unexpected behaviours over enough time.

The ARM Peripheral PDF describes things in terms of protecting things; 'barrier before writes' 'barrier after reads'. It does indicate those can be dropped when there's a guarantee nothing else will switch peripherals, and that's 100% safe when one has complete control over execution flow,, but that's by no means certain with a real-time, multi-core, multi-threaded, interruptible system which is what Linux / Raspbian is.

On the whole it doesn't seem to matter, mostly works, but there are some oddities with a Pi, unexplained SD card corruption and locking, unexplained PMIC failures, that I can't help but wonder if this isn't a part of the problem.

dl324
Posts: 109
Joined: Mon May 06, 2019 7:33 pm
Location: Pacific Northwest, USA

Re: Memory barriers for GPIO access

Thu May 23, 2019 4:17 pm

hippy wrote:
Thu May 23, 2019 3:46 pm
But what if I am accessing GPIO and something else then interrupts and accesses some other peripheral, or another parallel core or GPU accesses a different peripheral, and I don't have a barrier protecting things ?
Then you need to do what the datasheet said.

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Thu May 23, 2019 4:37 pm

dl324 wrote:
Thu May 23, 2019 4:17 pm
hippy wrote:
Thu May 23, 2019 3:46 pm
But what if I am accessing GPIO and something else then interrupts and accesses some other peripheral, or another parallel core or GPU accesses a different peripheral, and I don't have a barrier protecting things ?
Then you need to do what the datasheet said.
And, presumably, so should any library which accesses GPIO as they cannot guarantee against that either.

Would, that they don't, be deemed wrong or flawed in your opinion ?

I had a further thought; would it even be safe if read barriers were used ... ?

Code: Select all

My core                         | Some other core
                                |
ReadBarrier();                  |
:                               | ReadBarrier();
myStatus = *MY_PERIPHERAL_PTR;  | :
:                               | itsSatus = *ITS_PERIPHERAL_PTR;
ReadBarrier();                  | :
:                               | ReadBarrier();
return myStatus;                | :
                                | return itsStatus;
That's leading me to think it's not guaranteed to be safe on a multi-core system, only on single core.

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Fri May 24, 2019 10:04 am

hippy wrote:
Thu May 23, 2019 4:37 pm
That's leading me to think it's not guaranteed to be safe on a multi-core system, only on single core.
Everything I have now read on DMB and DSB on ARM leads me to believe that, not only does one need to use those, but, in a multi-core environment, one also needs to wrap those and the memory accesses with mutexes or locks.

But, while that will keep one's own code from causing problems, it can do nothing to prevent problems with or caused by other code which isn't using that mutex or lock.

I have said in the past that allowing every program to have unrestricted direct access to GPIO was a bad choice, allowing any program to change GPIO settings which other programs may be relying on, and particularly a problem if that program executes a poorly implemented clean-up routine.

It now seems to me that there are even more fundamental problems with allowing unrestricted access to GPIO, that the whole notion is flawed and seriously problematic for correct operation.

It would be great if someone could point to some documentation on this issue which demonstrably proves that assertion is incorrect. But I haven't found anything which does; everything I have read seems to only suggest it is correct.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22150
Joined: Sat Jul 30, 2011 7:41 pm

Re: Memory barriers for GPIO access

Fri May 24, 2019 12:29 pm

hippy wrote:
Fri May 24, 2019 10:04 am
hippy wrote:
Thu May 23, 2019 4:37 pm
That's leading me to think it's not guaranteed to be safe on a multi-core system, only on single core.
Everything I have now read on DMB and DSB on ARM leads me to believe that, not only does one need to use those, but, in a multi-core environment, one also needs to wrap those and the memory accesses with mutexes or locks.

But, while that will keep one's own code from causing problems, it can do nothing to prevent problems with or caused by other code which isn't using that mutex or lock.

I have said in the past that allowing every program to have unrestricted direct access to GPIO was a bad choice, allowing any program to change GPIO settings which other programs may be relying on, and particularly a problem if that program executes a poorly implemented clean-up routine.

It now seems to me that there are even more fundamental problems with allowing unrestricted access to GPIO, that the whole notion is flawed and seriously problematic for correct operation.

It would be great if someone could point to some documentation on this issue which demonstrably proves that assertion is incorrect. But I haven't found anything which does; everything I have read seems to only suggest it is correct.
And yet here we are, 6 years later, every thing seemingly working OK.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2158
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Memory barriers for GPIO access

Fri May 24, 2019 1:00 pm

A lack of synchronisation between cores (including ARM-to-VPU) can easily lead to classic read-modify-write non-atomic update problems, but fortunately there are a few factors in our favour:

1. High speed manipulation of GPIOs is more likely to be changing the drive on an output pin high or low, and that can be achieved with the atomic set/clear registers.
2. In most cases, changing the alternate function of a pin is done around the time the kernel is starting, and the GPIO/pinctrl driver has internal re-entrancy protection.

UPDATE: That was wishful thinking - the critical bcm2835_pinctrl_fsel_set appears to have no synchronisation (either within itself or around calls to it). Ho hum.

3. On the whole, the splitting of GPIOs between applications (2-27) and system (28-53) and the grouping of alt functions selectors (10 per register, so 0-9, 10-19, 20-29, etc.) tends to avoid simultaneous updates to any particular group.

But the lack of synchronisation is far worse for user-space libraries that go direct to the hardware. We should be switching as many of these as possible to the new gpiolib ioctl-based API, while beefing up the synchronisation within the driver.

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Fri May 24, 2019 5:37 pm

jamesh wrote:
Fri May 24, 2019 12:29 pm
And yet here we are, 6 years later, every thing seemingly working OK.
"Seemingly" being the operative word; absence of evidence is not evidence of absence.

"It has always seemed to worked so far" is hardly proof of there being no bugs or potential problems lurking. I've learned that the hard way, and so have others. There's a whole industry these days around finding bugs and flaws no one has so far spotted.

Open Source would likely not be half as reliable as it is if people weren't questioning what looked dubious to them. Many of the extremely serious bugs identified in recent times have been there for years, in software which otherwise "just worked fine" for millions of people, over billions of execution hours.

Maybe there's a problem, maybe there isn't. I saw a potential issue and asked if anyone could enlighten me. I'll be happy if there isn't an issue, if someone can provide evidence or reassure that it isn't. And I'll be happy if there is an issue and my noticing it ensures it ceases being an issue in the future.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22150
Joined: Sat Jul 30, 2011 7:41 pm

Re: Memory barriers for GPIO access

Fri May 24, 2019 8:29 pm

hippy wrote:
Fri May 24, 2019 5:37 pm
jamesh wrote:
Fri May 24, 2019 12:29 pm
And yet here we are, 6 years later, every thing seemingly working OK.
"Seemingly" being the operative word; absence of evidence is not evidence of absence.

"It has always seemed to worked so far" is hardly proof of there being no bugs or potential problems lurking. I've learned that the hard way, and so have others. There's a whole industry these days around finding bugs and flaws no one has so far spotted.

Open Source would likely not be half as reliable as it is if people weren't questioning what looked dubious to them. Many of the extremely serious bugs identified in recent times have been there for years, in software which otherwise "just worked fine" for millions of people, over billions of execution hours.

Maybe there's a problem, maybe there isn't. I saw a potential issue and asked if anyone could enlighten me. I'll be happy if there isn't an issue, if someone can provide evidence or reassure that it isn't. And I'll be happy if there is an issue and my noticing it ensures it ceases being an issue in the future.
Which is exactly why I asked PhilE to respond to your question, as he is an expert in this area.. I hope his answer has enlightened you.

I'm sure there is a problem, but one with a relatively low probability of happening.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

hippy
Posts: 5158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Memory barriers for GPIO access

Fri May 24, 2019 9:12 pm

jamesh wrote:
Fri May 24, 2019 8:29 pm
Which is exactly why I asked PhilE to respond to your question, as he is an expert in this area.. I hope his answer has enlightened you.

I'm sure there is a problem, but one with a relatively low probability of happening.
My apologies; I didn't catch the correct context of what you wrote. It sounded a bit like a slightly dismissive "it's been in every nuke we've ever built and none have spontaneously exploded", "people have been smoking our cigarettes for years and no one's complained", but obviously wasn't meant that way.

I do agree it is a 'once in a blue moon' issue. But we can't know for certain the probability, nor quantify the risk, or severity if it happens, know how many have had a problem and not realised it was through this, how many glitches or crashes could be put down to it. With 25 million devices out there, many running 24/7, there's going to be some chance of it happening, some people being adversely affected.

I'm not going to try and put a figure on what's acceptable and isn't.

I did find this past detailed discussion of the issue, where LdB reports actually running into the problem, and Ultibo indicate they used mutexes and/or locks to avoid it -

viewtopic.php?t=181306

Return to “C/C++”