SPI driver latency and a possible solution

[mahjongg]

I'm trying to get samples from an ADC at 200 samples/sec. Not fast, but my problem with the existing spidev driver is that the latency, so I can't control my sampling well enough and I was ending up with latencies > 4 ms.

Not to be pedantic, but this sounds like a job for the I2S rather than for any SPI interface, which lacks large buffers and/or other mechanisms to avoid latency problems. Unfortunately it seems that work on an I2S driver (for the new P5 GPIO header with I2S) is still ongoing.

Hi Martin

Thanks for all of your work on the low latency driver. I've followed it as best I can, but I think I'm going round in circles.

I'm trying to get samples from an ADC at 200 samples/sec. Not fast, but my problem with the existing spidev driver is that the latency, so I can't control my sampling well enough and I was ending up with latencies > 4 ms.

I think I've managed to apply your patch to 3.2.27 although I had to do some manual merging. (I think the github version had updated from my original version of raspbian.) Anyhow, I can still get data using spidev, although at the moment I'm just using usleep for my delay. (Presumably this is yet another source of my latency problem.)

I guess my question is, can I easily set up the driver to clock out data (and hence in from the ADC chip) every 4 ms, or do I need to use an onboard timer? If so, does anyone have any pointers?

Many thanks,
Jamie

Without any patches, I was able to sample the ADC on the Gertboard faster than that - in the region of 11K samples/sec without any kernel patches (and that was writing to the DAC at the same time - See the gertboard.c program in examples in wiringPi) So I'd be wondering if something else is going on in your code...

However if you want to sample it as a fixed frequency, then you have 2 choices (at least 2, anyway!) the first is that you enter the "big loop" type of programming style - and in that loop, you look for the time to be > the next sample time, and take the sample, the other, much easier IMO, but often harder to get to grips with, is to start a thread which runs concurrently with your program and that thread reads the time, (gettimeofday()) takes a sample, then waits (delayMicroseconds ()) for the time remaining until the next sample. You can simply keep the result in a global variable which the main code can read at any time, or even apply some digital filtering to it if needed, but still keep a global to allow the rest of your program to read the value.

(You could, if you were paranoid, have a function to return the result with some mutex guarding roound reading the varaible, but if the variable is a single 32-bit integer with just one process writing and many readers then there shouldn't be any issues)

WiringPi has some easy helper functions to help get you started with concurrent threads and mutexes, if needed... but maybe I need to do more examples as this sort of programming methodology can be quite confusing at first...

-Gordon

Well - that is a tricky question.

Also my potential to help you is limited by the fact that:
a) I am not an expert on SPIDEV (actually I have never used that one really)
b) you did not mention which ADC you are talking about

But in general I would say that a dedicated driver is be the best approach to reduce jitter.

If you are lucky then you may find your ADC in question in the list of drivers that are already in the linux kernel - I checked and found AD7266/65 and a few others in my kernel checkout (look in drivers/staging/iio/adc and drivers/iio/adc).

Obviously I have no idea how to use those (iio drivers) from user-space - you may have to look into that as well... (a quick hint for a driver that is there can be found here: http://www.at91.com/linux4sam/bin/view/ ... oAdcDriver)

There are other (theoretical) ways how you might do the same thing from user-space via SPIDEV, but these depend heavily on the ADC in question (especially on how you trigger sample) and also have some latency issues under some circumstances...

Assuming an 12 bit adc which is triggered with the SPI transfer of 2 bytes, an approach could be to configure the SPI clock to be acceptable to you - but you are very limited to the available sample rates - 250MHz/16/(2^prescaler) - and then just trigger a lot of reads - say 4096 bytes(or whatever SPIDEV allows) - and when this transfer is finished trigger the next "chunk". This would mean a latency every 2048 samples. But you could avoid that by having 2 threads where one trigger one such transfer while the other transfer is running and then swap roles - jitter may be minimal in such a setup as it would be primarily limited to the SPI driver itself.

Hope some of these ideas are helpful...

Martin

Gordon and Martin,

Thanks very much for your replies, and I should apologise for this slow reply. I don't get a huge amount of time to work on this.

Before I forget, my ADC is a Linear Technology LTC2452. It's a 16 bit ADC, capable of clocking at 2 MHz, but it is only capable of 250 samples/s, limited, I believe, by the conversion time, so I don't think that triggering a bunch of reads will work. I'll have a look inside the kernel, but I could not find it from within make menuconfig.

WiringPi looks promising. I haven't had time to check it out in detail yet, but shall try soon. The tone of both your posts suggests that I should really be looking at a multithreaded application. I think that's what I'll try next.

I suppose I was just expecting to have relatively easy access to a hardware timer and generate an interrupt from that, since I come from a microcontroller background. I guess I need to get used to life with a scheduler!

Thanks again, I will report back if (hopefully when) I get it working.

Jamie

From what I have seen in the datasheet I can think of the following "hack" from Userspace to minimize jitter:
* opening 2 SPIDEV devices (a,b) (open with O_NONBLOCK)
** one of them is your ADC (a)
** the other is a dummy device for timing purposes (b)
* configure both for speeds of say 1MHz (exactly 976562Hz =250MHz/2^8)
* schedule 2 byte reads on (a) asynchronously
* schedule immediately on (b) asynchronously a 2072 bytes transfer (=250MHz/2^8*0.017/8 - 0.017 is the sample interval you would like to see)
* using select() check that (a) is ready for read
* as soon as (a) is ready read the 2 bytes
* and trigger the next sequence of 2072 byte transfers on (b)

That way you pass on your timing to the SPI driver and you get the timing for free... (the SPI driver is executing the reading of data and subsequent scheduling the next few bytes in the kernel thread - and there typically without delays)

Please note that this "hack" could be possible with just a single thread, but I do not know if:
a) if SPIDEV allows for NonBlocking IO the way I think it should
b) if the SPI-driver of your choice can do this as exactly as you need - there still may be some jitter...
c) I have not really used SPIDEV in any way, so this is a bit theoretical
d) if you want to use the Stock kernel, then you will need to look into the SPI speed settings configured in arch/arm/mach-bcm2715/bcm2715.c and modify the above calculations with the correct clock-scaling (2^X)...

I would recommend that you measure the timing and the jitter between subsequent reads to see if it fits your requirements.
If not you will have to go the kernel route - it should not be too complex implementing that kind of thing in the kernel - make a copy of one of the other drivers in the iio directory and start coding those few steps:
* fill in a spi_message structure (spi_message_init())
* attach to it an spi_transfer structure with 2 bytes of read (potentially with a delay already set) - spi_message_add_tail(m,t)
* submit to spi queue - spi_sync(device,m)
* get the value and return it.

If you copy drivers/iio/adc/ad7266.c you need to basically modify the ad7266_trigger_handler, ad7266_probe and ad7266_remove functions and delete the functions that are unnecessary for the device of yours...
A better candidate to start could be: drivers/staging/iio/adc/ad7606_spi.c

Martin

For those who tried to apply the patch since the end of January, you should have seen a problem.

An update as been made to the kernel source, that give an error when trying to apply the patch:
https://github.com/raspberrypi/linux/co ... b613bce7d4

It seems that the commit adds only something that was already done in msperl's patch, so there is nothing new from this. However it is sufficient to make the patch fail.

Attached to this message is a "new" patch tailored for the current kernel source. It should result in almost the same file at the end.

I have not my Pi with me for some days, so for now, I only checked that it compiled, and that the source seems correct compare to the older patch.

If one of you wants to make a trial, and give me a feedback, I could correct it if anything is wrong.

If one of you wants to make a trial, and give me a feedback, I could correct it if anything is wrong.

Thanks, however ... one of the issues I have at the moment is that I'm writing some software to use an SPI device and I feel it's very unreasonable to expect people who may buy this device to have to upgrade their kernel, etc. (lets face it - it's hard enough to get people to do apt-get update/upgrade without then asking them to do more upgrades to get a new kernel )-:

So I'm wondering if it would be possible to build just the modules against the foundations released 3.2.27+ kernel so it could just replace the existing ones?

-Gordon

I feel it's very unreasonable to expect people who may buy this device to have to upgrade their kernel, etc. (lets face it - it's hard enough to get people to do apt-get update/upgrade without then asking them to do more upgrades to get a new kernel )-:

What do you mean by buy this device ? Buy a Raspberry Pi ?
The Raspberry Pi's goal is that people learn to program. So if one want to use some functionality that are not coming out of the box, then they will learn to do that.
The good point is that it is not difficult to do it, there are already lot of posts in this forum and pages on elinux.org to explain it. And people ready to help and explain if someone is in trouble.

So I'm wondering if it would be possible to build just the modules against the foundations released 3.2.27+ kernel so it could just replace the existing ones?

If it is possible, someone has to do it... which configuration do you need ?

For the record, my last patch (in fact msperl's patch updated to current kernel version) concerns the kernel 3.6, not the 3.2. You have to dig to the first pages of this topic to find back the 3.2.27+ kernel version. There was discussion to merge it officially, but I don't know where the discussion is now.

Also the SPI configuration depends on what you want to do with it... I have a kernel compiled to support a SPI bus at 10MHz with an MCP2515 running at 20MHz, some other people have 6MHz SPI clocks, some other are using completely different peripherals which would require a different setup, thus custom kernel/driver...
I don't know if there is a way to change at run time the SPI configuration written in the board definition (this the kernel). If not, you need to recompile it so that it does what your need, or use the same setup as someone else who provides you what he has compiled.

Zeta

I feel it's very unreasonable to expect people who may buy this device to have to upgrade their kernel, etc. (lets face it - it's hard enough to get people to do apt-get update/upgrade without then asking them to do more upgrades to get a new kernel )-:

What do you mean by buy this device ? Buy a Raspberry Pi ?

You trimmed a but much - I'm working on an SPI device to connect to the Pi that may be sold...

The Raspberry Pi's goal is that people learn to program. So if one want to use some functionality that are not coming out of the box, then they will learn to do that.
The good point is that it is not difficult to do it, there are already lot of posts in this forum and pages on elinux.org to explain it. And people ready to help and explain if someone is in trouble.

Sure - and I appreciate that ONE of the goals is to get people to program, but like it or not, people are producing commercial product based round the Pi. e.g. me - I've already produced my little ladder board while I'm selling (which I'll never retire on, however..) and that device is designed to get people to learn to program. This device I'm working on (can't say, NDA, sorry) may also be used to aid people to program too.

Lots of people are making and selling things for the Pi that cane be used in a learning environment, so I don't think there is any issues on that front.

There are also other commercial Pi projects going on that have nothing whatsoever to do with learning/teaching. That's fine too. By the end of this year there will be several devices with Pi's embedded inside them.

So I'm wondering if it would be possible to build just the modules against the foundations released 3.2.27+ kernel so it could just replace the existing ones?

If it is possible, someone has to do it... which configuration do you need ?

For the record, my last patch (in fact msperl's patch updated to current kernel version) concerns the kernel 3.6, not the 3.2. You have to dig to the first pages of this topic to find back the 3.2.27+ kernel version. There was discussion to merge it officially, but I don't know where the discussion is now.

Also the SPI configuration depends on what you want to do with it... I have a kernel compiled to support a SPI bus at 10MHz with an MCP2515 running at 20MHz, some other people have 6MHz SPI clocks, some other are using completely different peripherals which would require a different setup, thus custom kernel/driver...
I don't know if there is a way to change at run time the SPI configuration written in the board definition (this the kernel). If not, you need to recompile it so that it does what your need, or use the same setup as someone else who provides you what he has compiled.

Zeta

The device in question currently runs at 32MHz and while latency isn't an issue for it, it might be for some other devices..

It would just be nice if there were some clarity though - it seems some people are rushing towards 3.6, etc. but the foundation is still pushing 3.2.27 - do you think that one day I'll apt-get update/upgrade and get 3.6?

As for the configurations - I want the existing way of setting it to continue to work the way it currently does - ie. I currently have the abiltiy to select the channel and select the clock speed (without recompiling the kernel). I do not want that to change - all I'm after is a little less latency on 1 and 2 byte transactions. Zero inter-byte gap would be nice too, but I've a funny feeling that's a hardware limitation.

-Gordon

I'm working on an SPI device to connect to the Pi that may be sold...

It's clear now, didn't understood your previous post that way...

Like it or not, people are producing commercial product based round the Pi. e.g. me

No problem for me, except that if some additional work need to be done so that _your_ product works, I will not do it for you
I can only share there what I have done for mine, following the open source model.

It would just be nice if there were some clarity though - it seems some people are rushing towards 3.6, etc. but the foundation is still pushing 3.2.27 - do you think that one day I'll apt-get update/upgrade and get 3.6?

Early adopters are the one that debug the system so that the other can benefit from a stable release after some time. Its an old habit for me, to always work with the last version (old bug corrected, and new functionality available, but sometimes new bugs), and more fun, as there are a lot of things that fail and that you have to understand and correct!
My previous post had this goal : that some other people make a trial of the patch to check that it works correctly, so that then other people can use it if they need it.

I don't really know the plans about it, but logically it will finish to be pushed to raspbian.
However, using 3.6 or 3.2 should not change anything in the case of the SPI, so there is no need to change.

As for the configurations - I want the existing way of setting it to continue to work the way it currently does - ie. I currently have the abiltiy to select the channel and select the clock speed (without recompiling the kernel).

I would be interested to know how you do that without recompiling. Is it while loading the spi driver, or after it is loaded ?

I do not want that to change - all I'm after is a little less latency on 1 and 2 byte transactions. Zero inter-byte gap would be nice too, but I've a funny feeling that's a hardware limitation.

So if I understand well, your application already works with the current Raspbian kernel, with a bit of latency, so your are looking for msperl's low latency patch being merged to the Raspbian kernel ?

Zeta

I found the merge request posted by Martin 3 months ago : https://github.com/raspberrypi/linux/pull/147
But it concerned only the 3.6 branch.

As for the configurations - I want the existing way of setting it to continue to work the way it currently does - ie. I currently have the abiltiy to select the channel and select the clock speed (without recompiling the kernel).

I would be interested to know how you do that without recompiling. Is it while loading the spi driver, or after it is loaded ?

After it's loaded - using the existing (ie. pre Pi) access mechanism.

So in my wiringPi code, I have:

Code: Select all

static uint8_t     spiMode   = 0 ;
static int8_t     spiBPW    = 8 ;

and

Code: Select all

  if (ioctl (fd, SPI_IOC_WR_MODE, &spiMode)         < 0) return -1 ;
  if (ioctl (fd, SPI_IOC_WR_BITS_PER_WORD, &spiBPW) < 0) return -1 ;
  if (ioctl (fd, SPI_IOC_WR_MAX_SPEED_HZ, &speed)   < 0) return -1 ;

This is all published stuff (prior to the Pi)

I do not want that to change - all I'm after is a little less latency on 1 and 2 byte transactions. Zero inter-byte gap would be nice too, but I've a funny feeling that's a hardware limitation.

So if I understand well, your application already works with the current Raspbian kernel, with a bit of latency, so your are looking for msperl's low latency patch being merged to the Raspbian kernel ?

Zeta

[/quote]

That's basically it. And from what I've read here, what others are after too - people are already using the SPI drivers on the Pi - some with my wrappers in wiringPi, some natively and the 2 things that I've seen people have issues with is the inter-byte gap (of about 1-2 clock cycles), and the latency in starting an SPI transaction - which makes reading from an SPI device slower than its theoretically possible...

-Gordon

As for the configurations - I want the existing way of setting it to continue to work the way it currently does - ie. I currently have the abiltiy to select the channel and select the clock speed (without recompiling the kernel).

I would be interested to know how you do that without recompiling. Is it while loading the spi driver, or after it is loaded ?

Gordon explained howto modify the existing SPI entries. Its even possible to add and delete SPI definitions. The SPI module is using them when needed. So you could define them after loading the SPI module itself. Here an example for an OpenWRT router (http://lnxpps.de/wgt634u/wgt634u.c):

Code: Select all

static struct mcp251x_platform_data mcp251x_info = {
        .oscillator_frequency = 16E6,
        .board_specific_setup = NULL,
        .power_enable         = NULL,
        .transceiver_enable   = NULL,
};

static struct spi_gpio_platform_data wgt634u_gpio_spi = {
        .sck            = SPI_SCK_GPIO,
        .mosi           = SPI_MOSI_GPIO,
        .miso           = SPI_MISO_GPIO,
        .num_chipselect = 1, /* number of chip selects for spi gpio master */
};

static struct platform_device spi_gpio_device = {
        .name                   = "spi-gpio-wgt634u",
        //.name                   = "spi_gpio",
        .id                     = 1,                    /* Bus number */
        .dev.platform_data      = &wgt634u_gpio_spi,
};

static struct spi_board_info mcp2515_spi_gpio_board_info [] = {
        {
                .modalias               = "mcp2515",
                .max_speed_hz           = 10E6,
                .bus_num                = 1,
                .chip_select            = 0,
                .platform_data          = &mcp251x_info,
                .irq                    = 3,
                .mode                   = SPI_MODE_0,
                .controller_data        = (void *) SPI_CS_GPIO,
        },
};

/* Platform devices */
static struct platform_device *wgt634u_devices[] __initdata = {
        &wgt634u_gpio_leds,
        &spi_gpio_device,
};
static int __init wgt634u_init(void)
...
        platform_add_devices(wgt634u_devices, ARRAY_SIZE(wgt634u_devices));

        printk(KERN_INFO "WGT634U try to add SPI device ...\n");
        spi_master = spi_busnum_to_master(1);
        if (spi_master == NULL) {
                printk(KERN_ERR  "  can't determine SPI busmaster\n");
                goto FAILED;
        }
...
        if (spi_new_device(spi_master, mcp2515_spi_gpio_board_info) == NULL) {
                printk(KERN_ERR  "  WGT634U register MCP2515 SPI device failed\n");
        } else {
                printk(KERN_INFO "  WGT634U registered MCP2515 SPI device\n");
        }
...
}

This code looks like a board definition but it's compiled and used as a module.

The kernel docs (i.e. kernel/Documentation/spi/spi-summary) and the http://www.jumpnowtek.com/index.php?opt ... &Itemid=62 are a good start. I've tried to write a generic SPI device explorer but other things needed more attention ...

Regards

Gerd (not Gert)

I've tried to write a generic SPI device explorer but other things needed more attention

I've just done some work on adding and removing SPI devices at runtime.
I'm developing a framework for small LCD displays, and needed an easy way to change SPI devices when testing different settings.
First I was thinking of a generic SPI device manager, but in my case I have to pass platform_data as well, so I went the easy route and made it specific to my project.
The code could serve as a starting point for other projects.

Code: https://github.com/notro/fbtft/blob/master/spidevices.c
Doc: https://github.com/notro/fbtft/wiki/Module:-spidevices

The kernel docs (i.e. kernel/Documentation/spi/spi-summary) and the http://www.jumpnowtek.com/index.php?opt ... &Itemid=62 are a good start. I've tried to write a generic SPI device explorer but other things needed more attention ...

Regards

Gerd (not Gert)

Is that even possible? SPI is a sort of ad-hoc type of thing - make it u pas you go along - some devices are write only, some read only, some both... How would you be able to tell what's on the bus?

-Gordon

The kernel docs (i.e. kernel/Documentation/spi/spi-summary) and the http://www.jumpnowtek.com/index.php?opt ... &Itemid=62 are a good start. I've tried to write a generic SPI device explorer but other things needed more attention ...

Regards

Gerd (not Gert)

Is that even possible? SPI is a sort of ad-hoc type of thing - make it u pas you go along - some devices are write only, some read only, some both... How would you be able to tell what's on the bus?

-Gordon

Gordon,
maybe I wasn't precise enough - I'm talking about SPI slave devices, not about the master. IMO the main benefit from making a SPI device manager/explorer would be, that you don't have to recompile the kernel when you are going to connect a different SPI slave.

For vanilla SPI devices you could use ioctl like you mentioned and going on with spi-dev modul. But for other devices like MCP2515 for example you need more things like IRQ. A SPI explorer/manger could add the SPI slave specifc part.

Regards

Gerd

The kernel docs (i.e. kernel/Documentation/spi/spi-summary) and the http://www.jumpnowtek.com/index.php?opt ... &Itemid=62 are a good start. I've tried to write a generic SPI device explorer but other things needed more attention ...

Regards

Gerd (not Gert)

Is that even possible? SPI is a sort of ad-hoc type of thing - make it u pas you go along - some devices are write only, some read only, some both... How would you be able to tell what's on the bus?

-Gordon

Gordon,
maybe I wasn't precise enough - I'm talking about SPI slave devices, not about the master. IMO the main benefit from making a SPI device manager/explorer would be, that you don't have to recompile the kernel when you are going to connect a different SPI slave.

I don't currently recompile the kernel when I connect a different SPI slave device to my Raspberry Pi's.

Are we talking about the same thing here?

Or have I missed something obvious? e.g. the earler posts about compiling the speed into the kernel driver.

Right now, I have several different devices that I can connect to a Pi's SPI port - PiFace, Gertboard, some home-brew stuff, ATmega, RGB LED drivers device X and so on, and I connect these without recompiling the kernel, and they (almost) all work at different speeds.

For vanilla SPI devices you could use ioctl like you mentioned and going on with spi-dev modul. But for other devices like MCP2515 for example you need more things like IRQ. A SPI explorer/manger could add the SPI slave specifc part.

Regards

Gerd

I'm under the impression this thread was started precisely due to that chip (or some other CAN controller) so if we fix the kernel SPI latency then maybe there is no (or less) need to dive into the kernel to add in specific drivers...

-Gordon

I'm under the impression this thread was started precisely due to that chip (or some other CAN controller) so if we fix the kernel SPI latency then maybe there is no (or less) need to dive into the kernel to add in specific drivers...

You are right.
While Martin was working on making the MCP2515 driver work, he make this improvement on the SPI driver, and as it could benefit to other peripherals using the SPI, he opened up this thread to discuss it.

The point in recompiling the kernel with the MCP2515 is that CAN bus support is not available by default in Raspbian. So you need to activate it, add the drivers for your CAN controller, and configure your platform (SPI speed, controller crystal, IRQ, ...). There may be some other way to configure the SPI at run time, but as you still need to recompile the kernel to allow the CAN controller to be used, it is easy to change the platform configuration at the same time.

If it is not necessary for other devices, this is a good point.

Hi,
I have problems with the mcp2515- or the spi-driver, when I'm using two can's with high bus-load.

The commands

cangen can0 -g 50 -i -I 400
Strg-Alt-F2 (for the second terminal)
cangen can1 -g 50 -i -I 401

work fine,
when I reduce the times between the messages to

cangen can0 -g 1 -i -I 400
Strg-Alt-F2
cangen can1 -g 1 -i -I 401

after a short time, one of the can's stops sending messages.
And I cannot restart it with the same command.
The only way is to unload and load the module spi-bcm2708 or mcp2515

I tested the commands with two virtual cans and "candump any" on a third terminal
and there were no problems.

I am using the 3.6 kernel with the mcp2515 patch and the spi-patch.

Do you have any idea, what might happen here?
I am not good enough in programming to understand the code, but I can help you
by reading out values in the error state, ...

Thanks maddin1234

Hello Maddin,

You are writing you are using 2 CAN converters.
Can you give us more details about how they are connected ? What speed are you using ?
Do you use a single interrupt line shared between them, or 2 independent interrupts inputs ?
Can you check the results of the commands cat /proc/interrupts when one bus hangs ? and also ifconfig ?

I don't know if I can help on this, I did not tried with 2 CAN bus at the same time (I may be able to make a try next month, but don't have the hardware now for a second one). I had a hangout during reception, but that what corrected with the mcp2515 patch... (tested from 125kbps to 500kbps).

Zeta,

Hello Zeta,
thanks for trying to help me.

This is my configuration in the board-definition.
I used GPIO17 and GPIO18 for interrupt to keep the JTAG Pins unused.
(Maybe it is needed in future)

Code: Select all

#define MCP2515_CAN0_INT_GPIO_PIN 17
#define MCP2515_CAN1_INT_GPIO_PIN 18

.....

static struct platform_device bcm2708_spi_device = {
   .name = "bcm2708_spi",
   .id = 0,
   .num_resources = ARRAY_SIZE(bcm2708_spi_resources),
   .resource = bcm2708_spi_resources,
   .dev = {
      .coherent_dma_mask = DMA_BIT_MASK(DMA_MASK_BITS_COMMON),
   },
};

#ifdef CONFIG_SPI

static struct mcp251x_platform_data mcp251x_info = {
   .oscillator_frequency   = 25000000,
   .board_specific_setup   = NULL,
   .irq_flags              = IRQF_TRIGGER_FALLING|IRQF_ONESHOT,
   .power_enable           = NULL,
   .transceiver_enable     = NULL,
};

static struct spi_board_info bcm2708_spi_devices[] = {
   {
      .modalias = "mcp2515",
        .max_speed_hz = 10000000,
        .platform_data = &mcp251x_info,
      .bus_num = 0,
      .chip_select = 0,
      .mode = SPI_MODE_0,
   }, {
      .modalias = "mcp2515",
        .max_speed_hz = 10000000,
        .platform_data = &mcp251x_info,
      .chip_select = 1,
      .mode = SPI_MODE_0,
   }
};

static void __init bcm2708_mcp251x_init(void) {
   bcm2708_spi_devices[0].irq = gpio_to_irq(MCP2515_CAN0_INT_GPIO_PIN);
   printk(KERN_INFO " BCM2708 mcp251x_init:  got IRQ %d for MCP2515\n", bcm2708_spi_devices[0].irq);
   bcm2708_spi_devices[1].irq = gpio_to_irq(MCP2515_CAN1_INT_GPIO_PIN);
   printk(KERN_INFO " BCM2708 mcp251x_init:  got IRQ %d for MCP2515\n", bcm2708_spi_devices[1].irq);
   return;
};

#endif


lsmod gives this result:

Code: Select all

Module                  Size  Used by
mcp2515                 6100  0
can_bcm                11221  0
can_raw                 6052  0
can_dev                 9368  1 mcp2515
can                    23613  2 can_bcm,can_raw
spi_bcm2708             6916  0
snd_bcm2835            12939  0
snd_pcm                78130  1 snd_bcm2835
snd_page_alloc          5129  1 snd_pcm
snd_seq                53349  0
snd_seq_device          6454  1 snd_seq
snd_timer              19946  2 snd_pcm,snd_seq
snd                    58506  5 snd_bcm2835,snd_timer,snd_pcm,snd_seq,snd_seq_device
evdev                   9454  1
8192cu                486133  0


cat /proc/interrupts and ifconfig before any can traffic looks like this:

Code: Select all

           CPU0       
  3:       8213   ARMCTRL  BCM2708 Timer Tick
 16:          0   ARMCTRL  bcm2708_spi.0
 32:     191470   ARMCTRL  dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1
 52:          0   ARMCTRL  BCM2708 GPIO catchall handler
 65:        549   ARMCTRL  ARM Mailbox IRQ
 66:          0   ARMCTRL  VCHIQ doorbell
 75:          1   ARMCTRL
 77:       6361   ARMCTRL  bcm2708_sdhci (dma)
 80:        429   ARMCTRL  bcm2708_spi.0
 83:         19   ARMCTRL  uart-pl011
 84:      10517   ARMCTRL  mmc0
187:          0      GPIO  can0
188:          0      GPIO  can1
FIQ:              usb_fiq
Err:          0


can0      Link encap:UNSPEC  Hardware Adresse 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
          UP RUNNING NOARP  MTU:16  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:10
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

can1      Link encap:UNSPEC  Hardware Adresse 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
          UP RUNNING NOARP  MTU:16  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:10
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


Then I started sending can messages with:

Code: Select all

cangen can0 -g 1 -i -I 400 & cangen can1 -g 1 -i -I 401

here is the result from on both cans:
raspi can 0 is connected to channel 1, raspi can1 is connected to channel 2

Code: Select all

  ...
  88.866304 2  401             Rx   d 7 0E 14 87 64 D4 03 55
  88.866917 1  400             Rx   d 8 00 D0 74 66 12 00 4A 0D
  88.867496 2  401             Rx   d 5 91 7A 0C 47 5B
  88.868129 1  400             Rx   d 8 43 19 8A 27 72 57 A5 07
  88.868656 2  401             Rx   d 0
  88.869311 1  400             Rx   d 6 E7 D1 26 30 B3 58
  88.870006 2  401             Rx   d 8 DA 76 59 19 D3 24 4B 61
  88.870913 1  400             Rx   d 8 26 D1 C2 0B 46 E7 BC 39
  88.871522 2  401             Rx   d 8 07 08 22 6E EF 39 78 54
  88.872301 1  400             Rx   d 2 59 11
  88.872972 2  401             Rx   d 4 D9 55 4C 3E
  88.874252 2  401             Rx   d 7 20 01 2E 63 E8 76 5C
  88.874375 1  400             Rx   d 5 F0 DF 5C 08 FB
  88.875470 2  401             Rx   d 7 16 47 D3 00 47 C6 3B
  88.875653 1  400             Rx   d 8 87 6D E8 29 BC 01 62 7F
  88.876770 2  401             Rx   d 8 FA 3C 41 31 FC 7C 44 09
  88.876787 1  400             Rx   d 0
  88.877988 2  401             Rx   d 8 D3 51 EB 2B 03 53 81 3D
  88.879234 2  401             Rx   d 8 DA 89 5F 3E 11 67 08 22
  88.880408 2  401             Rx   d 2 8F 09
  88.881744 2  401             Rx   d 8 AF 59 69 03 5D F0 FD 24
  88.882892 2  401             Rx   d 3 8E 32 48
  88.884182 2  401             Rx   d 7 1D 36 20 15 3E 6F 79
  88.885398 2  401             Rx   d 6 C1 FF 38 44 17 C5
  88.886652 2  401             Rx   d 8 49 B5 0C 7B 38 C6 F3 4D
  88.887820 2  401             Rx   d 4 C0 B5 67 67
  88.889118 2  401             Rx   d 8 5E 7E 5B 02 48 4A 08 00
  88.890320 2  401             Rx   d 7 0A 02 7D 3C 1B 9C F3
  88.891554 2  401             Rx   d 8 78 8A BD 2A F5 25 53 6A
  ...

And this is what cat /proc/interrupts and ifconfig looks after the test:

Code: Select all

           CPU0       
  3:     517470   ARMCTRL  BCM2708 Timer Tick
 16:          0   ARMCTRL  bcm2708_spi.0
 32:     514060   ARMCTRL  dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1
 52:    1073048   ARMCTRL  BCM2708 GPIO catchall handler
 65:       1521   ARMCTRL  ARM Mailbox IRQ
 66:          0   ARMCTRL  VCHIQ doorbell
 75:          1   ARMCTRL
 77:       6499   ARMCTRL  bcm2708_sdhci (dma)
 80:    4522982   ARMCTRL  bcm2708_spi.0
 83:         19   ARMCTRL  uart-pl011
 84:      11404   ARMCTRL  mmc0
187:      40057      GPIO  can0
188:    1032992      GPIO  can1
FIQ:              usb_fiq
Err:          0


can0      Link encap:UNSPEC  Hardware Adresse 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
          UP RUNNING NOARP  MTU:16  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40057 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:10
          RX bytes:0 (0.0 B)  TX bytes:230601 (225.1 KiB)

can1      Link encap:UNSPEC  Hardware Adresse 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
          UP RUNNING NOARP  MTU:16  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:75126 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:10
          RX bytes:0 (0.0 B)  TX bytes:431722 (421.6 KiB)


Greetings maddin

Hi,
I made some more tests and found one thing that looks strange:


test 1 with -g 1 (from thread before)

Code: Select all

88.874252 2 401 Rx d 7 20 01 2E 63 E8 76 5C
88.874375 1 400 Rx d 5 F0 DF 5C 08 FB
88.875470 2 401 Rx d 7 16 47 D3 00 47 C6 3B
88.875653 1 400 Rx d 8 87 6D E8 29 BC 01 62 7F
88.876770 2 401 Rx d 8 FA 3C 41 31 FC 7C 44 09
88.876787 1 400 Rx d 0
88.877988 2 401 Rx d 8 D3 51 EB 2B 03 53 81 3D
88.879234 2 401 Rx d 8 DA 89 5F 3E 11 67 08 22
88.880408 2 401 Rx d 2 8F 09
88.881744 2 401 Rx d 8 AF 59 69 03 5D F0 FD 24
88.882892 2 401 Rx d 3 8E 32 48


test 2 with -g 1

Code: Select all

 
728.499377 1 400 Rx d 8 8D FD 06 25 28 49 C1 0C
728.500425 2 401 Rx d 8 6B E0 40 69 67 4D 31 60
728.500587 1 400 Rx d 8 DB AB 72 0C DB D5 A5 45
728.501673 2 401 Rx d 8 56 9D A3 1F 0F 70 3B 42
728.501693 1 400 Rx d 0
728.503247 2 401 Rx d 8 7E 05 02 0E 90 71 FD 26
728.504393 2 401 Rx d 4 4C B5 75 55
728.505713 2 401 Rx d 8 8B A7 CA 53 AC EC 48 5D
728.506923 2 401 Rx d 7 C0 62 F2 0B 30 53 42
728.508039 2 401 Rx d 0


test 3 with -g 2

Code: Select all

 
210.167305 1 400 Rx d 8 52 28 13 23 DF D7 00 4D
210.168882 2 401 Rx d 0
210.169467 1 400 Rx d 4 E1 01 D8 2F
210.171470 2 401 Rx d 1 13
210.171965 1 400 Rx d 2 8A 17
210.174639 1 400 Rx d 8 0C 38 0A 52 64 10 4E 3E
210.174658 2 401 Rx d 0
210.176865 1 400 Rx d 8 B3 F3 A3 0E E9 A2 64 23
210.179111 1 400 Rx d 5 20 5C 87 5F 12
210.181279 1 400 Rx d 0
210.183649 1 400 Rx d 8 F5 FE C9 39 47 67 67 12
210.185897 1 400 Rx d 8 25 11 7D 16 11 C3 A2 26
210.188147 1 400 Rx d 8 B0 16 1F 3D FE 67 AF 79


test 4 with -g 2

Code: Select all

 
68.054666 1 400 Rx d 8 9B 04 49 64 14 7E 11 4E
68.055219 2 401 Rx d 6 B1 52 2F 6D 4C C8
68.055888 1 400 Rx d 8 42 F2 7B 6B 0C 3D 16 6C
68.056417 2 401 Rx d 4 6F CC C1 45
68.057106 1 400 Rx d 7 F1 99 68 45 E2 3A BD
68.060883 2 401 Rx d 0
68.060896 1 400 Rx d 8 51 08 75 38 7E C0 2A 11
68.062283 2 401 Rx d 6 F2 96 52 4A CD 04
68.063473 2 401 Rx d 4 2B A4 82 40
68.064657 2 401 Rx d 2 22 03
68.066077 2 401 Rx d 8 C0 05 F6 7D 85 43 57 4C
68.067517 2 401 Rx d 8 5D D2 BB 6A CE 9F 5F 6D
68.068917 2 401 Rx d 8 BA 28 BE 1C 69 3C 88 5C
68.070217 2 401 Rx d 6 AC D9 06 4A DE 4E


In all of these traces, I have a message with DLC=0 and a very short time delta
to the message before.
The normal time between to messages is about 400-500us.
Directly before one can fails, there is one delta with 10-20 us.

With the options -L 0 or -L 8 for both cans, there no problem for more than one hour each.

Code: Select all

cangen can0 -g 1 -i -I 400 -L 0 & cangen can1 -g 1 -i -I 401 -L 0


(sending 0 byte long messages on both cans or sending 8 byte long messages on both cans)
Also two cangen commands on can0 worked fine.

Maybe a different problem is this here:

When I start it with -g 0, then only can0 messages are send

Code: Select all

cangen can0 -g 0 -i -I 400 & cangen can1 -g 0 -i -I 401


Also if I start them the other way round.

Code: Select all

cangen can1 -g 0 -i -I 401 & cangen can0 -g 0 -i -I 400


I tryed it with the virtual cans:

Code: Select all

cangen vcan0 -g 0 -i -I 400 & cangen vcan1 -g 0 -i -I 401


There I could see "the scheduler". With candump any, there were many messages on vcan0 and then many messages on vcan1 and so on.

All of these test were made with 500kBits.

But also with 125kBits and the command

Code: Select all

cangen can0 -g 5 -i -I 400 & cangen can1 -g 5 -i -I 401

there is the same behaviour
(-g 5 is needed, otherwise only one can is sending)

Code: Select all

133.036540 1  400             Rx   d 8 75 90 D5 6A 7D 02 11 19
 133.041566 2  401             Rx   d 8 E6 41 24 61 7D 5A 9B 76
 133.041788 1  400             Rx   d 8 4A 8E EA 7A 47 B4 22 1A
 133.046805 2  401             Rx   d 8 FF 13 AC 21 21 6F 93 75
 133.046819 1  400             Rx   d 5 BA 58 27 15 B3
 133.052060 2  401             Rx   d 8 3C 94 B3 32 40 57 07 24
 133.057292 2  401             Rx   d 8 69 33 15 70 9E 09 3A 58
 133.062467 2  401             Rx   d 7 CC F6 C4 02 CE CE 66
 133.067259 2  401             Rx   d 0
 133.073026 2  401             Rx   d 8 2C 35 D9 0C 31 1A 9F 53
 133.078249 2  401             Rx   d 8 B8 70 EB 65 21 56 6C 26
 133.083137 2  401             Rx   d 2 7E 04
 133.088264 2  401             Rx   d 0

I hope this helps a little bit in finding the problem.

Greetings Martin

Hello maddin,

The problem you have is probably more related to the MCP2515 driver and interrupt than to the SPI driver as it continues to work properly.

This seems close to the problem I had with the MCP251x drivers with the kernel 3.6. This seems to be related to the IRQF_ONESHOT flag that is needed with this kernel.

What I had was:
- kernel 3.2 + mcp251x was working
- kernel 3.6 + mcp251x worked some time and finished to hang
- kernel 3.6 + mcp2515 @ 500kbps at full load with 8 bytes messages works

52 * IRQF_ONESHOT - Interrupt is not reenabled after the hardirq handler finished.
53 * Used by threaded interrupts which need to keep the
54 * irq line disabled until the threaded handler has been run.

As I understands it, the ONESHOT flag doesn't allow to catch an interrupt happening while in the interrupt routine. So using the mcp2515x driver along with the spi low latency patch was enough to handle the message fast enough at 500kbps to avoid this particular case.

However, here you have 2 drivers. It doubles virtually the bus speed.
What may happen is that a irq is catched on the first mcp, while processing it, an irq is catched on the second mcp. It takes some time to finish to process the first mcp, then the spi switch to the second mcp and start reading the registers. Due to the time taken before starting this operation, a second message was already received in the buffer.
If, during the time between the reading is finished and the irq is release, a third message is received, the interrupt line of the mcp will be activated again, but the driver can not see it as it configured for one shot only. Then the interrupt line of that driver stays low, and no more falling edge can be detected, so this bus seems to stop.

You can see that the interrupt number is really low for the driver that stops working, while the other doesn't show any problems after : as it is therefore alone it is far more difficult to fall in the case where it doesn't have time to handle the interrupt before the next one.

What I propose you is to try one of this:
* test with kernel 3.2 + spi low-latency patch for 3.2 (available in the first pages of this thread) + mcp2515 driver
* put an oscilloscope to both "spi chip enable", and to both mcp2515 interrupts line, to see what I assumed above ( interrupt should go high while the spi enable, after the read of the buffer, and goes low again before the chip enable is disabled)
* test with kernel 3.6 + spi low-latency patch + mcp2515 driver + change irq flags to

Code: Select all

.irq_flags              = IRQF_TRIGGER_LOW | IRQF_ONESHOT,

Putting the irq in level mode and not edge mode should allow it to restart as soon as the irq handler is exited, even if the second interrupt happened while inside the handler and not catched because of the ONESHOT flag. Don't know if it will ever work, but one the principle, this should allow to avoid blocking the bus, even if some messages are lost.

We should continue this discussion on the can converter thread.

Zeta

Hello Zeta,
ok, I will try some of these things and will post the new informations on
the other thread:
viewtopic.php?f=44&t=7027&start=175

I don't have an 4 channel scope, but I can check the state of the interrupt line
when the error is there for example.

The big difference in the number of interrupts is because of the long time,
the test was running after the error occured.
When I stopped the test directly after the error, the number of interrupt-events
is nearly the same.

Greetings maddin

There is something interesting going on with RT Linux:
viewtopic.php?f=37&t=33809
which includes a description building Xenomai RT Kernel for RPi. Realtime kernels are bot faster,
but the the delay should be lower/deterministic. Read on:
http://code.google.com/p/picnc/source/b ... .hal_rpspi