High Speed Data Monitor

[olso4539]

Hi all,
I am starting to create a high speed data monitor in two parts.

An spi kernel driver
-Pulls data into a ring buffer
-Logs data to sd card on trigger
-Uses DMA to minimize cpu usage?
-Interrupts/triggers dma on MOSI low
-Uses 8 chip selects, might be tricky but possible to use dma to increment gpio
-4800 samples per second on 8 channels at 32 bits per sample (1.23 Mb/s)

A data monitor
-Plots data from the ring buffer
-Needs to be high speed (up to 38400 lines per second plotted)
-Background including Axes, update sub window with data
-Updates at maximum refresh rate possible with latest data in ring buffer
--When used with low speed data it would appear to scroll
--When used with high speed data would show frame captures
-Configurable from a configuration file
-OpenVG based? for maximum speed

[olso4539]

I have the data monitor partially implemented plotting rand() generated data. I still need to figure out OpenVG fonts to add axis labels. I have 8 channels, 1720 data points per plot updating at nearly one frame every two seconds. It's ~7300 lines per second with turbo. It uses about 25% cpu, and i think the bottle neck is in communication with the GPU. I started with the "rshapes" example posted by ajstarks on this forum and started modifying and stripping it down to create one pixel wide lines connecting the data points.

I'll post some source code soon, I'm hoping to get thoughts on improving the speed from the community and make this a continually evolving open resource. I also plan to open source the ADC hardware.

Update: source is now on github.
https://github.com/olso4539/HSDMPi

[joan]

Monitoring what?

[olso4539]

Joan,
Monitoring voltage data from an add on ADC board. Mostly strain gauge based transducers for force and pressure, but also thermocouples and thermistors for temperature. If written generically it could be used with any spi data.

It's mostly inspired by the KST plotting program, which is fantastic but not as fast as bare bones and it is not hardware accelerated on the raspberry pi (openGL not OPENGL ES or OpenVG). I considered modifying KST to run faster, and still might try some day. Big thanks to the developers of that software.
-olso4539

[olso4539]

I was experimenting with the code, and it turns out most of the time is spent on vguLine(). It seems odd that i can only get 7300 lines per second with 1000 Mhz Arm and 500 Mhz core. That's 137,000 arm instructions or 68,500 core instructions per line drawn. I'm looking into the released Broadcom drivers to see what is happening now. Maybe openGLES would draw lines faster?

[PiGraham]

I was experimenting with the code, and it turns out most of the time is spent on vguLine(). It seems odd that i can only get 7300 lines per second with 1000 Mhz Arm and 500 Mhz core. That's 137,000 arm instructions or 68,500 core instructions per line drawn. I'm looking into the released Broadcom drivers to see what is happening now. Maybe openGLES would draw lines faster?

It seems you asking about how to display data faster.
As you have realised passing data to the GPU is one limiting factor and you should try to send bigger datasets to the GPU at a go.

Don't try to plot each sample as you acquire it.

I'm not familiar with the vgu functions, and you mention an example with shapes that may be doing this, but perhaps consider using vguPolygon to plot lots of lines/samples at once. If you plot 100 points as a polygon your 7300 lines per second might be closer to 730,000 sample points per second.

To get a smooth scrolling display you only need update at ~25 - 50 Hz. Any faster and your screen just won't update to show it, and couldn't see the difference anyway.

If you are sampling at 38kHz you can buffer and plot 760 samples at a go (38k/50).

[olso4539]

PiGraham,
Thank you! Before i was appending each segment to a path, then drawing the path for each frame with vgDrawPath. Once you mentioned it I took another look at the available library functions and found vguPolygon() which can be used to load an array of points into a path instead of loading a line segment at a time. I was hoping that there was some way to load an array in at once. And with the VGboolean open option i can make it so that it doesn't connect the beginning and end.

I will make the changes, I have high hopes for a big improvement.

Update: Changing to vguPolygon has improved the speed to 310471 data points plotted per second! That's 8 channels, 648 points plotted per update at ~60 frames per second! Source is updated on github.
~/src/openvg/HSDMPi $ time ./HSDMPi 5000
points 648 n 5000 w 720 h 480

real 1m23.486s
user 0m20.880s
sys 0m7.210s


Any more suggestions for speed improvement?

[olso4539]

I've just realized that dynamic overclocking only kicks in under high cpu load and not high gpu load. On my pi this program is running around 25% cpu and the default arm and core speeds. It would be nice to kick up the gpu from 250Mhz to 500Mhz without force turbo and setting the warranty bit.

[PiGraham]

7500 to 310471 samples per second? That's a useful speedup!.

If you want to go faster still try timing the programs operations to see what takes up time.
If you are drawing a scrolling display you may benefit from defining a polygon only for the new points and draw the old points at a shifted location using the previously defined polygon.
There may be an optimum transfer size for GPU ops.
Are you processing the data at all?
Is there delay in the acquisition process?

[olso4539]

PiGraham,
I'm just starting to look into what is using most of the time. I'm still sitting at a low cpu load, so it's probably still gpu or communication to the gpu limited.

I had the same thought about drawing the new polygons and shifting the old ones, but i haven't gotten there yet in programming it. I also have to decide just how i want to pull in the data. I was thinking it would pull from a ring buffer, so although it's not efficient to re-plot every point, if i can re-plot the ring buffer fast enough it will scroll and then i don't have to figure out what was already plotted. Also, if the data outruns the plotting, it just skips points. So far I'm just happy to have exceeded my requirement for the number of points plotted per second, so I can start working on more details of the implementation.

I'm not currently processing the data at all, and it's just randomly generated until I write the spi driver to pull data from the ADC IC. I want to do a simple offset and scaling, a running average, max, min, standard deviation, least squares linear fit slope, and compute an fft and plot that as well. The fft plot update rate being slow would be acceptable (once per second?). Though the hello_fft source included with raspbian pumps fft's through the gpu at ludicrous speeds so i might try to go significantly faster.

There really isn't appreciable lag in the data acquisition. The chip that I'm going to connect to is self clocked and overwrites it's own data register 4800 times per second (times 8 of them). If the driver falls behind it would start dropping data. That operation will probably be handled by a spi kernel driver to the ring buffer i mentioned.

I actually introduced a bug when i switched to the vguPolygon() function. It seems like if I load too many in at once bad things happen. See here for more details.
http://www.raspberrypi.org/forums/viewt ... 33#p572533

[olso4539]

After failing to figure out the bug with vguPolygon I found a lower level function called vgAppendPathData to plot the data with. Using this method i was able to fix the bug, and increase the plotting speed. Plus it's just more direct to use. I'm now up to ~536k plot points per second. That's pretty impressive, given that it's calculating and drawing each line with overhead in the time it takes the cpu to do 1300 instructions.

I either introduced or revealed another bug to work out.
sudo vcdbg reloc is telling me "heap corruption detected" if I try to plot too many points at once, low gpu memory? Sometimes that causes the program to fail and not exit correctly, so some of the gpu memory doesn't get freed. Running the program again seems to clear out most of it, but there are still several residual entries. A reboot fixes this.

Increasing the GPU memory to 256M fixed the problem of it crashing, but it seems it's still not letting the gpu memory free correctly as vcdbg detects corruption after running it with 50,000 data points * 8 channels (it seems like this only happens with LOTS of plot points, I haven't seen any corruption under ~10k per channel). Of course, this number of data points plotted at once is absurd given that most screens aren't bigger than 1920 pixels wide and it's running on a $35 computer. I just like testing the limits.

Now it's on to setting up axis labels!

[olso4539]

I've updated the git repository to include three fonts and text rendering. It's a basic implementation, but it works. I would like to redo the implementation to allow the full list of installed fonts, but that is a ways off because i need to create/find a TrueType to openvg font converter. I would be glad to hear suggestions on getting this to work.

I'm starting to think about how I want to lay out the various plotting elements (plot window, menu bar, legend, axes...), so I've been looking over egl and dispmanx. As of right now my application (and the reference implementation for egl)...

Code: Select all

Creates an egl display connection  \                Creates a native window surface (dispmanx)
           |                        \                                     |
           V                         \                                    V
Creates an egl rendering context      \--->Creates an egl window surface from the native window surface and display conn.
          |                                                                        |
          |------------>  links the egl window to the rendering context <----------|

and links openvg to egl

I'm thinking then that there are two was to create my separate elements. Either create everything on the same window or create a native/egl window surface pair for each plot element and let dispmanx do the compositing in hardware. Separate pairs would require eglMakeCurrent to switch the window being drawn to. Is there another way to draw to several windows with openvg? Is there a lot of overhead involved with eglMakeCurrent?

[ajstarks]

I've updated the git repository to include three fonts and text rendering. It's a basic implementation, but it works. I would like to redo the implementation to allow the full list of installed fonts, but that is a ways off because i need to create/find a TrueType to openvg font converter. I would be glad to hear suggestions on getting this to work.

https://github.com/ajstarks/openvg has code for working with openvg and TrueType fonts (see: https://github.com/ajstarks/openvg/blob ... openvg.cpp)

[DidSomeoneSayPi]

I also have a use for this project and would like to contribute. I see the git has not been updated. Has there been any further work on it?