John_Uskglass
Posts: 3
Joined: Tue Sep 19, 2017 7:50 pm

Why is toggling GPIO pins not a race condition?

Tue Sep 19, 2017 8:20 pm

Hi everyone. I'm new here so I hope I'm posting this in the right place.

I've been working my way through the Baking Pi tutorials for bare metal programming on my Raspberry Pi, but I'm a bit confused about how the system for controlling the GPIO pins actually works. I mean, it does work, at least as far as I can tell on my board, but it also kind of looks like a race condition, so I don't really understand why it works. Here's what I mean.

My assumption going into it was that each pin would correspond to a bit in memory, which I would set and unset as required to switch it on and off. However instead, it appears to use a messaging system. Following this tutorial, I have learnt that there are separate on and off bits for each pin, and that by setting these bits, you essentially toggle the pin on and off. This works, but to me, this appears to be vulnerable to many race conditions.

For example, imagine I wanted to turn on the first and second GPIO pins one after another. I might first write 0b00000001 to the first byte of the on pins. Immediately after, I might write 0b00000010. This of course would overwrite my original bit with a 0. I could or them together of course, but this doesn't seem to be how the system works. In practise, I can set them one after another and it switches them both on. However I don't understand why. Equally the fact that these bits are used for passing messages implies that they get reset to 0. After all, if they didn't, then after switching a pin on and off again, both the on and off bits would be set to 1 and I wouldn't be able to do anything else. However if something that isn't my code is resetting those bits to 0, isn't there a risk that, for example, I could set the first byte to 0b00000001 to turn on the first pin, then whatever is responding to this would read the result and turn on the pin, I would set the byte again to 0b00000010 to turn on the second pin, and then the bits get reset before the next read?

I hope that makes sense. I'm sure there's something fairly basic I'm just misunderstanding here, but if anyone could help me to get my brain around this, it would be greatly appreciated. Many thanks. Laurie

User avatar
rpdom
Posts: 11793
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 6:07 am

The bits are not in memory. They are direct hardware locations that are mapped into memory space. They are just triggers. They are connected to the set and reset parts of the GPIO drivers, so writing a bit takes effect pretty immediately. You can write one at a time or all together it doesn't matter (just the former will take longer).

LdB
Posts: 582
Joined: Wed Dec 07, 2016 2:29 pm

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 8:22 am

John_Uskglass wrote:
Tue Sep 19, 2017 8:20 pm
I have learnt that there are separate on and off bits for each pin, and that by setting these bits, you essentially toggle the pin on and off. This works, but to me, this appears to be vulnerable to many race conditions.
It is a hardware R/S flip flop

https://en.wikipedia.org/wiki/Flip-flop_(electronics)

Look at the truth table

R = 1 S = 0 Q = 0
R = 0 S = 1 Q = 1
R = 0 S = 0 State unchanged

Look familiar ???????

R = 1 S = 1 is illegal but you can't write to both registers at once they are at different memory locations and the ARM can only write to one at any given time. The signal is cleared a short time after you write to it.

Even that is a simplification .. here is the blockup of what it really looks like and at the moment we have just discussed the upper triangle which is the output latch.
Image
So remember there is a lot of physical hardware between what you do on the CPU and the output pin.

The CPU architecture guys deal with all that stuff for you, and you don't ever need to worry about it, that is why they are paid the big bucks. Just read the sheet and do what it asks.

piras77
Posts: 148
Joined: Mon Jun 13, 2016 11:39 am

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 8:34 am

John_Uskglass wrote:
Tue Sep 19, 2017 8:20 pm
I'm sure there's something fairly basic I'm just misunderstanding here, but if anyone could help me to get my brain around this, it would be greatly appreciated. Many thanks. Laurie
Often there is a one-to-one relation between a bit and the peripheral port's pin value. And all the port's pins are set by a single peripheral register's word. Then, in order to set/clear a pin's value, you would need a read-modify-write cycle to update a pin's value. E.g. [PORT] = [PORT] ^ 0x4 to toggle the 3rd bit. And, as you mentioned, this includes a race condition.

However, the Pi's GPIOs pins are modified by two registers, not only one. There is a register to set a pin's value, another register to clear a pin's value (and a third register to read a pin's value). The set/clear registers are single shot. For example [SET] = 0x4 will set the pin that relates to the 3rd bit. All other pin's are not affected. That is, a zero-bit has no effect, only set bits have. This has the advantage that you can modify single bits w/o any side effects (as race conditions) to other bits.

See also pp.95 https://www.raspberrypi.org/app/uploads ... herals.pdf

LdB
Posts: 582
Joined: Wed Dec 07, 2016 2:29 pm

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 9:03 am

All of us have probably also over simplified this we probably need to make you aware of another thing.

The act of you executing the opcode does not make that signal come out on the IO port. The ARM cpu and the GPIO run at very different frequencies. If you physically counted the opcodes cycles for a port on and off opcode write you will come up with a figure in the many hundreds of MHz (like 250Mhz) that a PI can toggle a pin on the GPIO. Once you exceed the GPIO bus speed it gets very glitchy and you can get latchup on the IO block and there is always jitter between the opcode execution and the GPIO signal arrival.

It can't do anywhere near that sort of speed the GPIO bus puts physical limits on the toggle speed. Those limits are
Raspberry Pi 1 of 22.7Mhz, the Raspberry Pi 2 reaches 41.7Mhz and the Raspberry Pi 3 65.8 Mhz.

Background
https://github.com/hzeller/rpi-gpio-dma-demo

You will also find you will struggle to the Pi3 signal off the board at that speed without proper design as there isn't a huge drive power available and a lot of capacitance.

John_Uskglass
Posts: 3
Joined: Tue Sep 19, 2017 7:50 pm

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 8:50 pm

Hi everyone. Thanks for all the really detailed and informative answers! As you have probably deduced, I'm really not a hardware guy, but I find learning about how these things work at the level of the electronics fascinating. I really do appreciate you taking the time to explain this for me!

@rpdom From what I was thinking, I did wonder if these was just a completely different implementation to what I was imagining in memory, and indeed your answer confirms that. Thanks! Also why is your image a horrible spider??

@LdB Thanks for all the details, and particularly for the diagram. Not being much of an electronics guy, I'm still getting my brain around how all of that works, but the flip-flop totally makes sense to me, and answers my basic question :). I feel I understand now, even though I would probably still freak out if you gave me a breadboard and a bunch of wires and told me to do it myself!

Your follow up about frequencies is also really interesting. So to check I've understood correctly, you're saying that the GPIO pins are able to respond to these signals at a lower frequency that the Pi's CPU can issue them, and thus it is still possible to introduce bugs if I toggle a pin on and off multiple times within a single cycle of the GPIO bus, which is possible because the cycle of the CPU is so much faster, right? But also if I've understood, there is no problem with issuing multiple signals to different pins, or indeed repeating the same instruction to a single pin (e.g. on, on, on) within a single GPIO bus cycle, right?

@piras77 Thanks for the link to the Broadcom docs. Yes I can definitely see how the implementation makes sense, and makes life a lot easier. This gives me the why as well as the how!

User avatar
rpdom
Posts: 11793
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Why is toggling GPIO pins not a race condition?

Wed Sep 20, 2017 9:02 pm

John_Uskglass wrote:
Wed Sep 20, 2017 8:50 pm
@rpdom Also why is your image a horrible spider??
Nightshade isn't horrible. He's a little softy really. Those are my wife's hands he's on.

We have a little deal. When she gets a new spider I can buy a new Pi. :-)

I'm a bit behind on the Pi count...

dwelch67
Posts: 840
Joined: Sat May 26, 2012 5:32 pm

Re: Why is toggling GPIO pins not a race condition?

Thu Sep 21, 2017 1:32 am

When making a chip for pins (pads with solder balls) like this you use I/O pads which is a hybrid mostly analog thing but has some signals that tie into the logic produced by the chip vendor or that is more purchased IP whatever. we see this I/O pad in an above drawing as well as in the broadcom document. there will be some discrete control signals that indicate say input vs output, pull up vs pull down, open collector, whatever was designed in. might have input interrupt sensors, etc...

Although there are interfaces often called serdes, that are gigabit or ten plus gigabit (getting above 10 lately with copper no less, driven by broadcom no less, but up to now it is mostly 10 gig per interface and if you want more you use multiple sets of interfaces) gpio and other things like that are not fast interfaces, they are way slower than the internals of the chip, for many reasons the voltages inside the chip in the wee bitty transistors is quite small relative to what we use on external interfaces, the signals are fairly clean, when you start to put big pads with big wires and mix them on a board you get noisy, you need more power to drive through the noise, etc and it becomes expensive in various forms of the word to go fast. So when buying/choosing the IP for your I/O pads you pick some speed, and then you design the digital logic portion of the chip that interfaces that to not outrun that, sometimes it is the software folks that have to keep to that limit, for example look at a number of the new microcontrollers using arm cortex-ms. you might be able to get the core to 40 or 80 or 120 or more MHZ, but the flash is often limited to like 20, the sram can usually go cpu speed, but the peripherals other than usb are often in the 20 or so range and you have to configure your clocks to not exceed any of these numbers. Likewise some chips let you pick your clock rate on the uart or spi or i2c and sometimes possible that you can outrun the I/O pad and the signal degrades to eventually not work, the analog side of things. Just like our brain may work at one speed, but our ear drums have a very limited mechanical range, outside that they just dont work. same with vocal cords and our vision. And our arms and legs and fingers. All of our human I/O is really slow compared to what is going on inside. chips are no different for pretty much the same reason, simple physics.

for output, think of the registers as being a single 32 bit wide memory location, but they are actually discrete flip flops tied to the I/O pad if the pad is set for output. some chips you have a single output register that you have to read-modify-write to change one or more bits, lazy logic designers, or not realizing that we are using this interface for discrete signals not for a parallel port. some like the pi's chips have a set register and a clear register that way you can set or clear more than one at a time. but you get into the situation that if you want to do some setting and clearing "at the same time" you cant (there is at least one mcu design where you can do that half the register is the set the pin bits and half is the clear the pin bits). you simply have to design the interface or write the bits in certain orders if you need to set and clear "at the same time".

something microcontrollers very often do as they are more starved of pins than this chip, is multiplex or mux pins, as you can see in the broadcom docs each of the "GPIO" pins are multifunction. some might be gpio n and uart 0 tx and spi clk or something. you have a mux which is like a tv input selector the various functions are brought to this switch and a software controlled selection determines which signal actually gets connected to the I/O pad or gets to control it, the others signals just go off into the ether. on the input side there is usually a similar scheme particularly with a chip like this where say there is more than one pin that can be used for uart rx, you dont want both pins directly connected all the time esp if that is not the function selected for that pin. one could be high the other low (driven from the outside because we might not be using one as the uart rx) and internally there is a fight and someone loses. so a good design leaves those inputs for the unused functions floating or driven by an alternate source. we could try that some day make two pins inputs to a common peripheral and drive one high and low and see if the chip burns up...

particularly for "GPIO" pins a chip design will power up with those set as inputs, so long as the voltages are safe you can drive the inputs whatever way you want. one board/system design may choose gpio x to be an input where another design an output if we defaulted the pins to outputs the one where it is an input we might fight with the other side of the trace one chip driving high one low and heat happens somewhere, if not controlled the it becomes smoke. so as programmers knowing the board design we switch outputs like led controls as part of the initialization and then we can blink them on and off or whatever.

back to the speed thing, the uart logic as a whole may be in a slower clock domain, with a boundary (driving through a school zone). or they may just leave it up to the programmer to not try to set or clear the pins too fast. usually somewhere in the middle it makes no sense whatsoever to have the uart, spi, gpio, etc peripherals clocked at 1ghz or whatever the cpu is running, even 250mhz is just crazy no need to burn that much heat for something your use case, and just the nature of how things work you cannot change that fast anyway. Most of the time it takes you a few instructions per state change on average depending on what you are trying to do, so right there you can cut the clock rate to that block by a few times. with the various things we know about in this chip being on the 250mhz, I bet 250mhz is the clock for most of the chip, peripherals, data busses, etc. and only the cpu core is at the higher rate, this is not uncommon and generally wise. then as mentioned as you get to the edge of the chip you have to step down from that to not hose the signal quality.

so we dont have any race conditions that we know about. technically if you set more than one bit in the gpio set register, yes there is a place where there is a race, in the digital domain, they are all clocked in parallel, no problems, but near the edge when it becomes analog there will be a difference both in routing and in just physics of the subtle differences in the pads from each other and if you can measure it they will come out of the chip at different times, all other factors held constant. but could you measure that? do you care? will it affect bit banging spi, etc? very doubtful...but when in doubt like with bit banging spi or i2c, just make sure one happens before the other with different writes to the I/O registers if you cannot tolerate a race or if one signal has to change before another. keeping signals parallel is why we dont do that, our hard drives use serial interfaces that have to run 16 or 32 or 64 times faster than the parallel interface it replaced, but that is easily tolerated, much easier to pick the ones and zeros out , even if recovering the clock, from a serial interface than a parallel over some long ribbon, etc. pcie for example is a number of x1, x2, x4, x8, x16 individual serial interfaces and the data is sprayed across them in packets using a physical layer protocol so that we can move 2x, 4x, 8x, 16x the amount of data that a single serial interface can handle, but not do it purely parallel where isolating time of a bit period becomes very difficult...so slow that down dozens of times and you get gpio which might have a race if you set two bits "at the same time" but unlikely you could detect it without a certain cost scope or other...

sorry so long hope this helps...

LdB
Posts: 582
Joined: Wed Dec 07, 2016 2:29 pm

Re: Why is toggling GPIO pins not a race condition?

Thu Sep 21, 2017 4:20 am

John_Uskglass wrote:
Wed Sep 20, 2017 8:50 pm
Your follow up about frequencies is also really interesting. So to check I've understood correctly, you're saying that the GPIO pins are able to respond to these signals at a lower frequency that the Pi's CPU can issue them, and thus it is still possible to introduce bugs if I toggle a pin on and off multiple times within a single cycle of the GPIO bus, which is possible because the cycle of the CPU is so much faster, right? But also if I've understood, there is no problem with issuing multiple signals to different pins, or indeed repeating the same instruction to a single pin (e.g. on, on, on) within a single GPIO bus cycle, right?
Spot on, you have got it and David has explained that is true of many other function blocks. It's quite common now because CPU have gotten so fast that software will be required to have control written into it to make sure it can't exceed the slower block access. The designers will try to make sure exceeding isn't fatal but the behaviour will often just become unpredictable.

John_Uskglass
Posts: 3
Joined: Tue Sep 19, 2017 7:50 pm

Re: Why is toggling GPIO pins not a race condition?

Thu Sep 21, 2017 3:58 pm

@rpdom I'll take your word for it! If I were going to share a house with a spid like that though, I think I'd demand at least one Pi per leg ;).

@dwelch67 Wow, thanks for all the amazing info there! I'm definitely going to need to read through that a few more times before it all sinks in, but I appreciate you taking the time to write it all out for me. I'm learning a lot here.

@LdB It's definitely interesting to think about how increased CPU clock speeds have made everything else seem proportionally slower. It's funny because as a software developer, I deal with this reality all the time when it comes to memory access speeds, which have become proportionally slower and slower, but I never stopped to think about how it affected other peripherals too.

User avatar
rpdom
Posts: 11793
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Why is toggling GPIO pins not a race condition?

Thu Sep 21, 2017 7:18 pm

John_Uskglass wrote:
Thu Sep 21, 2017 3:58 pm
@rpdom I'll take your word for it! If I were going to share a house with a spid like that though, I think I'd demand at least one Pi per leg ;).
If that were the case, I'd be looking at getting around 400 Pis instead of the 30 or so I have at present.

colinh
Posts: 94
Joined: Tue Dec 03, 2013 11:59 pm
Location: Munich

Re: Why is toggling GPIO pins not a race condition?

Fri Sep 22, 2017 1:59 am

rpdom wrote:
Thu Sep 21, 2017 7:18 pm
John_Uskglass wrote:
Thu Sep 21, 2017 3:58 pm
@rpdom I'll take your word for it! If I were going to share a house with a spid like that though, I think I'd demand at least one Pi per leg ;).
If that were the case, I'd be looking at getting around 400 Pis instead of the 30 or so I have at present.
Hang on! Spiders don't have 13 legs!?

User avatar
rpdom
Posts: 11793
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Why is toggling GPIO pins not a race condition?

Fri Sep 22, 2017 4:55 am

colinh wrote:
Fri Sep 22, 2017 1:59 am
Spiders don't have 13 legs!?
Correct. :lol:

I only have about 30 Pis at present because I haven't kept up with the spider count. If I did I'd have more anyway. But if it were one Pi per leg I would have to be aiming for the number I quoted. Actually probably more, because it was based on spiders bought, not spiders we still have.

Return to “Bare metal”

Who is online

Users browsing this forum: No registered users and 3 guests