eth0 RX/TX Queues

[JohnDuffy]

Hi

On my Raspberry Pi 4 with Ubuntu 20.04 64-bit, I have the following queues on eth0...

Code: Select all

john@node1:~$ ls /sys/class/net/eth0/queues/
rx-0  tx-0  tx-1  tx-2  tx-3  tx-4

I would be very grateful for any assistance with a couple of questions...

1.) With Ubuntu and Raspberry Pi OS both being Debian derivatives, is this queue arrangement the same on Raspberry Pi OS?

2.) Is the split between TX/RX queues configurable? I have an application which consumes a lot of data over eth0 and returns much less. So, if possible I would like to have rx-0 to rx-4 and a single tx-0. I would then plan to bind each rx queue to a specific CPU using Receive Side Scaling and Receive Packet Steering. I'm happy to edit the genet driver source code recompile the kernel if necessary to reconfigure the queues.

By the way, I'm only using Ubuntu because I need a 64-bit OS. As soon as Raspberry Pi OS 64-bit is out of beta I will be switching.

Kind regards

[trejan]

1.) With Ubuntu and Raspberry Pi OS both being Debian derivatives, is this queue arrangement the same on Raspberry Pi OS?

Yes. It uses the same Broadcom GENET driver.

2.) Is the split between TX/RX queues configurable?

The driver is capable of multiple TX and RX ring queues but the hardware parameters for GENET V5 only has 4 TX priority queues. You can try adjusting that value and recompiling the kernel but I suspect it is a hardware limitation.

[JohnDuffy]

Fabulous!

I did have a look through the driver code but didn't spot the hw params.

I'm going to swap the tx and rx buffer parameters around, compile a new kernel, and see what happens...

Thanks for the pointer.

[JohnDuffy]

A couple of questions please...

In bcmgenet.c is the RX_BUF_LENGTH a per RX queue parameter? And is there a hardware limit to this size?

A pointer to any Broadcom documentation of the NIC would be much appreciated.

[JohnDuffy]

Update:

I recompiled a kernel with the rx/tx queue hw params in bcmgenet.c reversed.

Before this change I had the these queues: rx-0, tx-0, tx-1, tx-2, tx-3, tx-4

I now have: rx-0, rx-1, rx-2, rx-3, rx-4, tx-0

I haven't tested Receive Side Scaling (RSS), Receive Packet Steering (RPS), Receive Flow Steering (RFS) yet, but so far so good. Nothing broken.

[trejan]

A pointer to any Broadcom documentation of the NIC would be much appreciated.

No public documentation.

I haven't tested Receive Side Scaling (RSS), Receive Packet Steering (RPS), Receive Flow Steering (RFS) yet, but so far so good. Nothing broken.

You need to try using those queues. The default configuration doesn't use them at all and the driver just assumes the hardware config values are correct.

[JohnDuffy]

Hi

Would you mind expanding on "the default configuration doesn't use these queues"? Is is possible to change the configuration to do so?

I have made the appropriate changes to /proc/sys... and /sys/class... to implement RSS and RFS, but there is no apparent change to network functionality or performance. Which is a shame, because I was hoping to benefit from improved data locality for my Linpack benchmarks. It would appear, as you state, that these queues are not used.

[fruitoftheloom]

.
The RaspiOS64 Beta is extremely stable and will get even better going forward as it is supported by the RPT / RPF..

...it is after all based of Debian Buster Stable ARM64.