robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

CM4 concurrent access to multiple I2C busses

Tue Jul 27, 2021 9:36 pm

I've incorporated the CM4 into an industrial controls system where I am using 4 of the CM4's I2C busses (i2c1, i2c4, i2c5 and i2c6). I have a separate C++ thread to communicate with each i2c bus in order to speed transmissions and get better throughput on each bus.

Each thread works fine when enabled on its own (only one bus seeing transactions at a time), but when all 4 are enabled, about 1% of the transactions fail with a "Connection timed out" (110) error.

The entire program is in C/C++ using the ioctl, linux/i2c.h drivers to read/write from the bus. Any suggestions on what to look for to fix this issue?

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 concurrent access to multiple I2C busses

Tue Jul 27, 2021 11:09 pm

what does "ls -l /sys/class/i2c-adapter/" output?
what pins are each i2c bus on?

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 1:03 am

Here are the pins that each bus is connected to:
I2C1: GPIO2 & GPIO3
I2C4: GPIO6 & GPIO7
I2C5: GPIO10 & GPIO11
I2C6: GPIO22 & GPIO23

All busses work as configured when tested separately, and work ~99% of the time when together.

Relevant parts of /boot/config.txt are:
dtparam=i2c_arm_baudrate=200000
dtoverlay=i2c6,baudrate=100000
dtoverlay=i2c5,pins_10_11
dtoverlay=i2c4,pins_6_7,baudrate=400000
dtoverlay=i2c1

Output of "ls -l /sys/class/i2c-adapter/":
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-0 -> ../../devices/platform/soc/fe205000.i2c/i2c-11/i2c-0
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-1 -> ../../devices/platform/soc/fe804000.i2c/i2c-1
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-10 -> ../../devices/platform/soc/fe205000.i2c/i2c-11/i2c-10
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-11 -> ../../devices/platform/soc/fe205000.i2c/i2c-11
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-4 -> ../../devices/platform/soc/fe205800.i2c/i2c-4
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-5 -> ../../devices/platform/soc/fe205a00.i2c/i2c-5
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-6 -> ../../devices/platform/soc/fe205c00.i2c/i2c-6

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 3:53 am

yeah, those 4 i2c busses are indeed seperate busses

my guess is that its just a kernel bug, perhaps grabbing a shared mutex

aBUGSworstnightmare
Posts: 3467
Joined: Tue Jun 30, 2015 1:35 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 6:51 am

Where does I2C0/I2C10 come from? It' not in OP's list of busses and for sure muxed.

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 7:26 am

robotmike wrote:
Wed Jul 28, 2021 1:03 am
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-0 -> ../../devices/platform/soc/fe205000.i2c/i2c-11/i2c-0
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-10 -> ../../devices/platform/soc/fe205000.i2c/i2c-11/i2c-10
lrwxrwxrwx 1 root root 0 Jul 27 18:02 i2c-11 -> ../../devices/platform/soc/fe205000.i2c/i2c-11
aBUGSworstnightmare wrote:
Wed Jul 28, 2021 6:51 am
Where does I2C0/I2C10 come from? It' not in OP's list of busses and for sure muxed.
i2c-10 and i2c-0 are likely from the pinmux i2c mux driver
basically, you tell the linux kernel each set of pins the i2c controller can be routed to, via the pinmux driver
and linux then creates multiple virtual i2c master ports (i2c-0 and i2c-10)
when you try to use a virtual master, it will re-mux the controller to the right set of pins, and then issue the command on the true master (i2c-11)

it lets you access more buses at once, at the cost of having to share a single controller

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11632
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 8:42 am

Are you reading, writing, or both?

All the I2C controllers share one common interrupt on the ARM, but that should be level triggered so multiple devices interrupting should all be serviced in turn.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 4:08 pm

In general, the busses are reading and writing. Three of the busses are probably at > 50% utilization, with two being a roughly-equal mix of reading and writing, and the other one being mostly reading. The fourth bus is probably only 10% utilized, mostly reading.

I'm also gravitating to an interrupt issue, since the I2C waveforms look normal at the time of the error, but the driver is indicating a timeout, which my guess is a missed interrupt.

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Wed Jul 28, 2021 8:45 pm

As a side note, I also have

Code: Select all

dtparam=i2c_vc=on
in my /boot/config.txt to enable the I2C for the camera (CSI). I'm not using the camera at the time of testing.

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Thu Jul 29, 2021 5:09 am

I tried adding

Code: Select all

dtoverlay=i2c-bcm2708
to /boot/config.txt, which switched i2c1 over to the older BCM2708 driver. Testing this configuration fixed any timeout issues from i2c1, yet i2c4-6 still show timeouts. Here's an exerpt from running "tail -f /var/log/syslog":

Code: Select all

Jul 28 21:58:00 raspberrypi kernel: [  313.034540] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:10 raspberrypi kernel: [  323.674747] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:34 raspberrypi kernel: [  347.275205] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:35 raspberrypi kernel: [  348.635122] i2c-bcm2835 fe205c00.i2c: i2c transfer timed out
Jul 28 21:58:39 raspberrypi kernel: [  352.235162] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:42 raspberrypi kernel: [  355.195196] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:46 raspberrypi kernel: [  359.275240] i2c-bcm2835 fe205c00.i2c: i2c transfer timed out
Jul 28 21:58:51 raspberrypi kernel: [  363.995336] i2c-bcm2835 fe205c00.i2c: i2c transfer timed out
Jul 28 21:58:52 raspberrypi kernel: [  365.355302] i2c-bcm2835 fe205c00.i2c: i2c transfer timed out
Jul 28 21:58:52 raspberrypi kernel: [  365.355325] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:54 raspberrypi kernel: [  366.795318] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:58:58 raspberrypi kernel: [  371.435359] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:59:01 raspberrypi kernel: [  373.915381] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Jul 28 21:59:04 raspberrypi kernel: [  376.875399] i2c-bcm2835 fe205800.i2c: i2c transfer timed out
Does anyone know how to change i2c4 thru i2c6 over to the BCM2708 driver?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11632
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CM4 concurrent access to multiple I2C busses

Thu Jul 29, 2021 6:20 am

robotmike wrote:
Thu Jul 29, 2021 5:09 am
Does anyone know how to change i2c4 thru i2c6 over to the BCM2708 driver?
You look at what the i2c-bcm2708 overlay does (https://github.com/raspberrypi/linux/bl ... verlay.dts) and duplicate it for the other ports.
The "combine" parameter on the i2c1 overlay does the same thing, so it might be as easy to add that option to the other overlays instead.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Thu Jul 29, 2021 5:55 pm

I've modified i2c4-overlay.dts (and for i2c5, i2c6) by adding:

Code: Select all

	...
	{
	...
	fragment@3 {
		target = <&i2c4>;
		__dormant__ {
			compatible = "brcm,bcm2708-i2c";
		};
	};

	__overrides__ {
		pins_0_1 = <0>,"=1!2";
		pins_22_23 = <0>,"!1=2";
		baudrate = <&frag0>, "clock-frequency:0";
		combine = <0>, "!3";
	};
but when "combine=0" is specified, the driver refuses to load (the i2c device does not appear under /sys/class/i2c-adapter/). The .dtbo works OK without the combine=0 option, and it works fine with the stock i2c1 file.

I also tried the separate i2c-bcm2708 overlays with the same unsuccessful result.

Any suggestions shy of Kernel modification? Thanks for your help!

robotmike
Posts: 7
Joined: Tue Jul 27, 2021 9:20 pm

Re: CM4 concurrent access to multiple I2C busses

Thu Jul 29, 2021 8:54 pm

The problems above were running the stock 5.10.17-v7l kernel. From the raspberrypi/linux GitHub, I cloned the current 5.10.y branch, compiled and installed it, and do not see the timeout issue above (using the BCM2835 driver). So I think this issue is solved for now.

For reference, the new kernel is 5.10.52-v7l, compiled and installed using the instructions here: https://www.raspberrypi.org/documentati ... uilding.md

Return to “Interfacing (DSI, CSI, I2C, etc.)”