msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Tue Oct 23, 2012 8:14 am

The one_message is the "RT-thread" that handles and sleeps, while the _interrupt is the corresponding interrupt handler, that in the DMA case wakes up the RT-worker thread.

For the interrupt mode the interrupt handler needs to feed additional data to the FIFO (and also read the data), so it is a bit more complicated...

Maybe the IRQ handler functions should be named _irqhandler instead for better understanding...

Ciao, Martin

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Wed Oct 24, 2012 5:52 pm

Hi!

here a patch relative to the git 3.6.y branch.

One issue at this very moment: the mcp251x driver does not load on my system right now - interrupt issues, but that seems to be a different issue with the 3.6 kernel and the config/driver...

Ciao, Martin

P.s: kernel message for the IRQ looks like this:

Code: Select all

[ 4191.728405] genirq: Threaded irq requested with handler=NULL and !ONESHOT for irq 195
[ 4191.728489] mcp251x spi0.0: failed to acquire irq 195
Attachments
spi-latency-branch3.6.y.patch.bz2
patch relative to git branch 3.6.y
(8.38 KiB) Downloaded 701 times

uski
Posts: 13
Joined: Sat Oct 27, 2012 12:41 pm

Re: SPI driver latency and a possible solution

Sat Oct 27, 2012 12:43 pm

I think I'll try to use this mod with a PCF8833 LCD. I may make a framebuffer driver for it, or I might use it in userland - not sure yet.

This is great stuff. Thanks for sharing, I'll be monitoring the thread actively and I'll let you know if I use it.

zia_7575
Posts: 11
Joined: Thu Oct 18, 2012 6:39 am

Re: SPI driver latency and a possible solution

Sat Nov 17, 2012 10:44 am

Hey Martin,

I am using raspberry pi with ADS1298 chip .. I have applied your spi_low_latency patch. Interrupt works fine for me.. but whenever I try to use dma, I do not receive data.. plus there is no timeouts. Please help me to find the data and how can I trace it ?

Regards,
Zia

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Sat Nov 17, 2012 11:14 am

Does your driver for the chip supply DMA-allocated memory ranges?
(i.e fill in the dma_rx/dma_tx fields in the spi_transfer structures - you only get those values if you allocate the pages via alloc_coherent.

If you do not fill it in, then it should fall back to IRQ driven mode.

My guess is that these values filled in right now are the SAME addresses as the pointers you use in your driver itself for access to the memory - these are not IDENTICAL to addresses that DMA is using (there is another Address translation layer between the Kernel and the real bus-address), so this can result in memory modified that is not yours - it could be modifying a user process memory or other parts of the kernel structure/code...

BTW: DMA only makes sense if you transfer lots of data in one "go" - say a full page of data (=4096 bytes) ... if you transfer less than 64 bytes (Size of the FIFO buffer) then the IRQ version works just as well.

For IRQ mode you will get one IRQ per 64 bytes transferred - so if you transfer 4096 bytes you will get 64 Interrupts. DMA-mode would handle this with only 1 IRQ (transfer finished) instead... So the real advantage is really only for high volume transfers (say to SD card or other devices that can be the source or destination of lots of data...)

But this only applies to the situation if you request >64 byte transfers in a single operation - I am not 100% clear if this is the case or not...

Ciao,
Martin

zia_7575
Posts: 11
Joined: Thu Oct 18, 2012 6:39 am

Re: SPI driver latency and a possible solution

Mon Nov 19, 2012 10:13 am

yep . I didn't provide the dma_tx and dma_rx. can u guide me about how to use alloc_coherent function ?
and I have to call it from userspace or spidev.c ? Any example ?

U r rite about the use of DMA. I am not transferring too much data in single operation ..I just want to use DMA for learning purpose.:)

regards,
Zia

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Mon Nov 19, 2012 2:20 pm

So, as you use "small" transfers, the normal "interrupt-driven" mode should be more than good enough for your use cases...

As for DMA buffer allocation, look at the mcp251x.c CAN bus driver (drivers/net/can/mcp251x.c)
This one has the module load-option to allocate memory from DMA: mcp251x_enable_dma=1

Here the relevant code-section from that module to allocate the memory:

Code: Select all

        /* If requested, allocate DMA buffers */
        if (mcp251x_enable_dma) {
                spi->dev.coherent_dma_mask = ~0;

                /*
                 * Minimum coherent DMA allocation is PAGE_SIZE, so allocate
                 * that much and share it between Tx and Rx DMA buffers.
                 */
                priv->spi_tx_buf = dma_alloc_coherent(&spi->dev,
                                                      PAGE_SIZE,
                                                      &priv->spi_tx_dma,
                                                      GFP_DMA);

                if (priv->spi_tx_buf) {
                        priv->spi_rx_buf = (priv->spi_tx_buf + (PAGE_SIZE / 2));
                        priv->spi_rx_dma = (dma_addr_t)(priv->spi_tx_dma +
                                                        (PAGE_SIZE / 2));
                } else {
                        /* Fall back to non-DMA */
                        mcp251x_enable_dma = 0;
                }
        }

        /* Allocate non-DMA buffers */
        if (!mcp251x_enable_dma) {
                priv->spi_tx_buf = kmalloc(SPI_TRANSFER_BUF_LEN, GFP_KERNEL);
                if (!priv->spi_tx_buf) {
                        ret = -ENOMEM;
                        goto error_tx_buf;
                }
                priv->spi_rx_buf = kmalloc(SPI_TRANSFER_BUF_LEN, GFP_KERNEL);
                if (!priv->spi_rx_buf) {
                        ret = -ENOMEM;
                        goto error_rx_buf;
                }
        }
Note that this only works inside Kernel-space (it is not exposed to user-space), so you will have to write your own driver (or modify the spidev driver, which I doubt will work, because it translates the user-space addresses to kernel space addresse and these are not DMA mapped)

Ciao,
Martin

zia_7575
Posts: 11
Joined: Thu Oct 18, 2012 6:39 am

Re: SPI driver latency and a possible solution

Wed Nov 21, 2012 11:41 am

Hey,

I return the pointer to userspace via following code in spidev.c

Code: Select all

int spidev_kmem(struct file *filp, struct vm_area_struct *vma)
{
	//printk(KERN_ALERT "ZIA SPIDEV_KMEM STARTING");
        int ret;
        long length = vma->vm_end - vma->vm_start;
	//printk(KERN_ALERT "ZIA length of Memory = %ld",length);
        /* check length - do not allow larger mappings than the number of
           pages allocated */
        if (length > NPAGES * PAGE_SIZE)
	{
		printk(KERN_ALERT "ZIA SPIDEV_KMEM ERROR 1 code:%d" , EIO);
                return -EIO;
	}

        /* map the whole physically contiguous area in one piece */
        if ((ret = remap_pfn_range(vma,
                                   vma->vm_start,
                                   virt_to_phys((void *)kmalloc_area) >> PAGE_SHIFT,
                                   length,
                                   vma->vm_page_prot)) < 0) {
		printk(KERN_ALERT "ZIA SPIDEV_KMEM ERROR 2 code:%d" , ret);
                return ret;
        }
	//printk(KERN_ALERT "ZIA SPIDEV_KMEM END");

        return 0;
}

/* character device mmap method */
static int spidev_mmap(struct file *filp, struct vm_area_struct *vma)
{
	//printk(KERN_ALERT "ZIA SPIDEV_MMAP STARTING");
        /* at offset NPAGES we map the kmalloc'd area */
        if (vma->vm_pgoff == NPAGES) 
	{
                return spidev_kmem(filp, vma);
        }
	printk(KERN_ALERT "ZIA SPIDEV_MMAP END with ERROR! code:%d" , EIO);
        /* at any other offset we return an error */
        return -EIO;
}
but this is not the address which DMA used. How can I map bs->dma_buffer to above memory ?
I don't know, it's a valid question or not.. :( I just want to return the pointer of DMA buffer to userspace...

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Wed Nov 21, 2012 4:12 pm

Sorry, but what you want is unfortunately not possible.

I have actually been looking for ways to get the VideoCore address (which is the DMA address of tx_dma/rx_dma) but that would be a hack and might change with different versions.
Also a driver with such a "hack" in place might never get included with the mainline kernel in the first place.... - that is unless it is an interface that is usable for other purposes as well...

That is a bit of a project and my priorities currently lie elsewhere.

The easiest if you really want to use DMA is writing a driver of your own and acuireing a DMA-able page of 4096 bytes (or two) and schedule those pages one after another.
then also provide a callback function with your SPI request, which then does copy the data of the filled page to user-space and then reschedules itself. That is obviously only valuable for continuous reads without any other intervention... The problem then is that in case of your ADC chip, the SPI bus will need to drive the sampling with the "normal" clock...
If you have to do:
* signal the ADC to take a sample
* read a few bytes from ADC (say 3 bytes for 24 bit resolution)
* wait some time
* loop

Then DMA can not help you in the first place, because DMA can not do those control-loops with external signalling itself. If you want to controll via SPI BUS frequency, then your frequency-options are limited to 250MHz/(2^X) (unless you overclock your system, which will also increase core frequency - but then if the system gets too "hot", then it will scale down, which will make your SPI bus sample rate not "constant"...)

Hope this helps...

Ciao,
Martin

firefox0815
Posts: 1
Joined: Sun Dec 09, 2012 10:43 pm

Re: SPI driver latency and a possible solution

Sun Dec 09, 2012 10:50 pm

hi,

i just wanted to thank you and give a report from an user of your patch. i have a project running. i try to use spi to drive led stripes with tm1812 chips. before the latency was great and incalculable. since i use the realtime patch the jitter between two 8 bit words at 4mhz is really low (i am not exactly shure but i think) less than 1 ysec. so the stripes are now working (most of this... but i think its a problem with the power supply). both dma or irq mode are working quite well so far.

cheers

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Mon Dec 10, 2012 6:19 am

@Firefox: Good to hear!

Note that there is a new "module" in the latest version of the 3.6.y tree that is called "BCM2708DMAHelper" that provides information on Memory Mapping from userspace all the way to Bus Addresses.

With that we can (probably) do DMA from/to userspace without any copying...

This may possibly mean changes to the (userspace) SPIDEV driver in addition to the driver for the bcm2708 to make it work with zero copying!

I will need to find some time to test/implement this idea...

Ciao,
Martin

gorge_alan_sw
Posts: 50
Joined: Tue Nov 27, 2012 12:54 pm

Re: SPI driver latency and a possible solution

Wed Dec 26, 2012 7:08 am

Hi Martin,

I really appreciate your hardwork. I am also trying to use MCP2515 with Rpi.
Just one thing I went through your interrupt based SPI patch.

But i did not found routine which is using --- external interrupt from MCP2515 ----- to indicate a packet is received. Is this patch is regarding -- interrupt when -- external interrupt pin is set high by the MCP2515 or normal SPI tx/rx interrupt :?: I am not that much intrested to Dig inside kernel but one thing i know if we patch to kernel then it will affect internal master driver of SPI ("Controller Driver").

Or We have to implement the -- external interrupt from MCP2515 --- inside the protocol driver for the MCP2515.

What should be the right MCP2515 driver to use with your patch. I found one but it uses spi_sync (which is blocking version & can add to latency ) is it fine to use. Or RPI have its own protocol driver for the MCP2515 :?:
http://lxr.free-electrons.com/source/dr ... 51x.c#L956


Please suggest, Any reply will be appreciable.

//Allan

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Wed Dec 26, 2012 11:20 am

Well, the SPI driver is essentially generic and can also get used for other devices like ADCs/Network drivers/ SD cards,...

The mcp2515 part you are talking about:
there is one mcp251x driver that comes with the standard kernel, but that one shows issues with the 3.6 kernel, where at some point the interrupt does not get handled any more by the mcp251x driver.
there is an alternative driver mcp2515 that does handle it gracefully...

See the discussion here: http://www.raspberrypi.org/phpBB3/viewt ... =44&t=7027 (last few pages on the issues seen lately)
See also http://lnxpps.de/rpie/ for a quick way to get can working and there also is a link to the mcp2515 driver.

In any case you will need to build your own kernel to make it work...

Finally: I have been in contact with the developer of the mcp251x driver and he said, he would look into the issue after Christmas... This would make it easier to get it back into the main kernel tree...

Ciao,
Martin

User avatar
[email protected]
Posts: 1979
Joined: Tue Feb 07, 2012 2:14 pm
Location: Devon, UK
Contact: Website

Re: SPI driver latency and a possible solution

Sat Dec 29, 2012 9:40 pm

Does anyone know if there has there been any progress in getting this into the kernel distributed by the foundation yet?

I have a little project on the go - working well, but the latency is halving the speed I ought to be getting...

Cheers,

-Gordon
--
Gordons projects: https://projects.drogon.net/

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Sun Dec 30, 2012 5:57 pm

It still has not been merged - I have asked already..

You can request it to get merged at:
https://github.com/raspberrypi/linux/pull/147

maybe if multiple people ask, then it gets merged...

Martin

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: SPI driver latency and a possible solution

Mon Jan 07, 2013 1:37 am

@msperl - many thanks for the work on the SPI driver

It works quite well with my SD card (when my dodgy pin adapter works)

Some editing of the bcm2708.c file was required, namely setting the max bus hz to a more useful 20MHz (turns out at 15.6MHz due to the power-of-2 divisor) and assigning mmc_spi to spi0.0:

Code: Select all

static struct spi_board_info bcm2708_spi_devices[] = {
        {
                .modalias = "mmc_spi",
                .max_speed_hz = 20000000,
                .bus_num = 0,
                .chip_select = 0,
                .mode = SPI_MODE_0,
        }, {
                .modalias = "spidev",
                .max_speed_hz = 500000,
                .bus_num = 0,
                .chip_select = 1,
                .mode = SPI_MODE_0,
        }
};
Note that Chip Enable flaps around quite a bit on my scope in interrupt-driven mode - so using DMA mode is a must for SD card transfers (and is faster anyway). The data lines are reasonably clean at 15.2MHz but my connecting ribbon cable is about 3cm long. Any longer and it will likely corrupt data.

If I pass processmode=3 to the driver then I create a kernel OOPS on module insertion at the first data transfer (card detection). I can't get the stack trace at the moment because my SD connector gets in the way of the UART pins. The screen console displays a brief dump but drops into kdb - the OOPS is due to "unable to handle virtual paging request @ <address>" in bcm2708_transfer_one_message.

I believe it is due to

Code: Select all

static int __init bcm2708_spi_init(void)
{
        /* range check for processmode */
        if ((processmode<0) || (processmode>3)) { 
	        processmode=1; 
	}
	return platform_driver_probe(&bcm2708_spi_driver, bcm2708_spi_probe);
}
module_init(bcm2708_spi_init);
processmode should be range checked for 0..2. Also should we not fail at probing with an out of range parameter rather than override? I see a further check in bcm2708_spi_probe but this is never fruitful because it's already overridden.

If I treat everything right I get these performance figures with DMA mode:

Code: Select all

[email protected] ~ $ sudo hdparm -tT /dev/mmcblk1

/dev/mmcblk1:
 Timing cached reads:     2 MB in  2.11 seconds = 972.46 kB/sec
 Timing buffered disk reads:   4 MB in  4.15 seconds = 987.54 kB/sec
With approx 63% cpu usage on a pi at stock speeds. Not bad!

Idreeszaman
Posts: 2
Joined: Mon Jan 07, 2013 11:34 am

Re: SPI driver latency and a possible solution

Mon Jan 07, 2013 12:02 pm

@msperl
I have applied your patch.I am using your dma process mode. When i send one message i successfully receive that MSG in loop back but when i try to send two or more messages, kernel gets stuck.When I set the SPIDLEN register to one message length. Then it does not stuck but I couldn’t receive message. I have allocated memory by dma_alloc_coherent in a protocol driver.

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: SPI driver latency and a possible solution

Mon Jan 07, 2013 12:53 pm

Argh - I found that the driver was actually falling back to interrupt mode when I was doing my testing earlier, which is why in /proc/interrupts I found 20,000 spi interrupts being triggered per second.

The mmc-spi driver determines if a device is dma-capable by looking at the dma_mask parameter of the spi master's device structure.

I went ahead and added dma_mask as DMA_BITS_COMMON (basically 0xFFFFFFFF) to the bcm2708.c file (and now mmc_spi thinks the spi master is DMA capable) but now I get DMA timeouts in the manner that ldreeszaman describes.

I can see the DMA interrupts for bcm2708_spi firing at a rate of a dozen or so per second but then for no reason they stop.

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Mon Jan 07, 2013 2:10 pm

Hi!

I remember that I have testes MMCs as well and they were working fine.
Only thing is that the DMA mode does not lend itself to the way that SPI communication works for SD-Cards.

The main reason is that you have to do polling on state until the card signals an "OK" - data written or similar.
And that results in 1byte DMA getting done - in that case interrupt driven mode works much better...

But I remember it was working when I tested it. (and I have tested MCP2515-CAN-Controller, ENC28J60 network adapter and an SD card).

I may need to look into the driver again...

Martin

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: SPI driver latency and a possible solution

Mon Jan 07, 2013 8:43 pm

msperl wrote:Hi!

I remember that I have testes MMCs as well and they were working fine.
Only thing is that the DMA mode does not lend itself to the way that SPI communication works for SD-Cards.

The main reason is that you have to do polling on state until the card signals an "OK" - data written or similar.
And that results in 1byte DMA getting done - in that case interrupt driven mode works much better...

But I remember it was working when I tested it. (and I have tested MCP2515-CAN-Controller, ENC28J60 network adapter and an SD card).

I may need to look into the driver again...

Martin
SD cards can transfer up to 2048 bytes (plus token + crc) in one SPI transaction if you issue a read_multiple_block command or write_multiple_block command. It is however true that some of the "control" command/response are 8bit responses only.

The mmc_spi driver knows about this and uses longer transfers when reading and writing. This means for the bulk of a normal transaction (read 2k at a time, write 2k and then wait for busy) the DMA can be running with no CPU interaction.

gorge_alan_sw
Posts: 50
Joined: Tue Nov 27, 2012 12:54 pm

Re: SPI driver latency and a possible solution

Tue Jan 08, 2013 4:22 pm

Hi msperl,

First of all i appreciate your good work for RPi beginners on CAN.
I have some questions Please suggest.

1> DMA for SPI is used for the tx or rx of data ?

2> what is the exact API to communicate with SCPI DMA driver in your patch ?

3> Also if i use INT pin of the MCP2515 to indicate that packet is recived on MCP2515 buffer
Then should i use DMA to read data from SPI or direct read from driver.
Will it be time consuming if we read all data from interrupt handler.

Ideally how shall we read data for the received data ?


4> Please suggest right MCP2515 driver to use for this which i can use as a base to build upon.
Are these the right driver to use in interrupt mode As per your previos post on my query.

>> there is an alternative driver mcp2515 that does handle it gracefully...
http://lnxpps.de/rpie/
http://clientes.netvisao.pt/anbadeol/mcp2515.html

5> Also what is the right SPI controller driver file name & path to which your patch have to be applied.


Being a newbie please help me in this design stage. Any reply will be appreciable.


//Allan

Zeta
Posts: 72
Joined: Wed Dec 12, 2012 9:51 pm

Re: SPI driver latency and a possible solution

Tue Jan 08, 2013 6:45 pm

Hello Allan,

You can find a lot of answer on the topic about the CAN controller : http://www.raspberrypi.org/phpBB3/viewt ... =44&t=7027

To use the patch of msperl, simply apply it like that :

Code: Select all

cd /opt/raspberrypi/linux_3.6.y
wget http://www.raspberrypi.org/phpBB3/download/file.php?id=1492 -O spi-latency-branch3.6.y.patch.bz2
bunzip2 spi-latency-branch3.6.y.patch.bz2 
patch -p1 << spi-latency-branch3.6.y.patch
Where "/opt/raspberrypi/linux_3.6.y" is the git cloned directory of the kernel.
Check the link to the file, so that it correspond to the last one that was posted here.

Hope this helps.

msperl
Posts: 325
Joined: Thu Sep 20, 2012 3:40 pm

Re: SPI driver latency and a possible solution

Tue Jan 08, 2013 9:47 pm

@George: I believe the easiest place to start your MCP2515 experience is with the following page:
http://lnxpps.de/rpie/

Things work with the kernel provided there for quick first steps - everything else will come along later..
(Building new kernels, applying patches,...) You may experience some "latency issues" and more, but essentially things are working for prototyping purposes...

As for DMA in general: the SPI driver I have programmed can do DMA for RX AND TX. The only requirement is that the higher level driver is (for now) providing additional pointers with the addresses in DMA space in addition to the ones with normal addresses. (Those DMA pointers need to get allocated in a special manner - see some discussions earlier in this thread).

As for your INT question in regards to the mcp2515: I really would leave it to the mcp251x or mcp2515 drivers and not write your own high-level drivers. (I have to admit that I am currently using the mcp2515 driver...) As for using Interrupt-driven or DMA driven: the way that the Chip in the RPI is working (especially the size of the fifo buffers) and taking into account the amount of data that gets transferred in one go when talking to the MCP2515, there is no difference whatsoever between DMA and normal interrupt-driven mode. The data transfers are typically 14 byte in size for a message, which fit in the FIFO of the RPI chip and thus the advantage of DMA is negligable (actually the setup code for DMA is a bit more complicated, so it is probably using a few more cycles to achieve the same thing...) DMA really is more of Interest for situations where you need to transfer more than 64 bytes, because the interrupt driven mode triggers an interrupt whenever the FIFO gets "empty" so about every 48 to 64 bytes. If you need to transfer 8KB then DMA would trigger 1 interrupt white the interrupt driven mode would trigger 128 times, which is wasting CPU cycles...

Again: for you case: Start with the simple steps and then progress from there...
(The thread http://www.raspberrypi.org/phpBB3/viewt ... =44&t=7027 may also give you good advice)

Ciao,
Martin

gorge_alan_sw
Posts: 50
Joined: Tue Nov 27, 2012 12:54 pm

Re: SPI driver latency and a possible solution

Thu Jan 10, 2013 4:29 am

Thanks Martin & Zeta for your help.

//Allan

jb1981
Posts: 2
Joined: Thu Dec 20, 2012 12:11 am

Re: SPI driver latency and a possible solution

Sat Jan 12, 2013 12:47 am

Hi Martin

Thanks for all of your work on the low latency driver. I've followed it as best I can, but I think I'm going round in circles.

I'm trying to get samples from an ADC at 200 samples/sec. Not fast, but my problem with the existing spidev driver is that the latency, so I can't control my sampling well enough and I was ending up with latencies > 4 ms.

I think I've managed to apply your patch to 3.2.27 although I had to do some manual merging. (I think the github version had updated from my original version of raspbian.) Anyhow, I can still get data using spidev, although at the moment I'm just using usleep for my delay. (Presumably this is yet another source of my latency problem.)

I guess my question is, can I easily set up the driver to clock out data (and hence in from the ADC chip) every 4 ms, or do I need to use an onboard timer? If so, does anyone have any pointers?

Many thanks,
Jamie

Return to “Interfacing (DSI, CSI, I2C, etc.)”

Who is online

Users browsing this forum: No registered users and 14 guests