OpenVG program crashes randoml, code or faulty unit?

[pawelduda]

Hello

I have explained what happens in readme.md of ajstark's OpenVG implementation fork here:
https://github.com/pawelduda/openvg

The problem is, I have wasted countless hours on this issue which really slows down the development of my program. Sometimes it crashes after 10 minutes, the other times after 40 minutes. I don't think I have ever gotten with the program running without crash for longer than 90 minutes. The problem is, the final program is expected to run all day.

Would there be a kind soul with the B+ model who could clone the repository and test it on his/her Raspberry Pi? That way I could decide whether it's a hardware or software problem. I made it so that setup process is as minimal as I could.

[Paeryn]

From the screenshot you gave it looks like there's been some memory corruption in the path data for one (or more) of the text glyphs. I can't see anything glaringly obvious in the code (apart from that you didn't declare the generateImageFromJpeg function in the shapes.h header file). I've had your program running for over an hour now and it's still working fine, I'll leave it running overnight and let you know if it's still running then.
Update:
It had locked up the whole RPi (display had frozen, can't ssh in) but no on-screen corruption after about an hour and a half.

[pawelduda]

Hey Paeryn, thank you for your feedback! I think it's enough to conclude that the issue lies somewhere within code.

I've had the same issue as you a few times. Meanwhile I left hello_tiger program running over night and it worked flawlessly.

I'll attempt to gradually rewrite my code to use OpenVG API directly instead of ajstarks' wrapper and hopefully I'll manage to track the issue.

[jamesh]

Hello

I have explained what happens in readme.md of ajstark's OpenVG implementation fork here:
https://github.com/pawelduda/openvg

The problem is, I have wasted countless hours on this issue which really slows down the development of my program. Sometimes it crashes after 10 minutes, the other times after 40 minutes. I don't think I have ever gotten with the program running without crash for longer than 90 minutes. The problem is, the final program is expected to run all day.

Would there be a kind soul with the B+ model who could clone the repository and test it on his/her Raspberry Pi? That way I could decide whether it's a hardware or software problem. I made it so that setup process is as minimal as I could.

Things like this are very very very unlikely to be hardware issues. Possibly GPU software issues, but again unlikely (the Khronos libraries all pass the comprehensive khronos test cases). They are almost always software problems at the user end.

[pawelduda]

I just needed someone to confirm this because perspective of wasting my time over something that I had no influence on seemed grim.

Anyway, I think I've managed to fix some things. I've removed the two font typefaces that were put in the code by the author of those libraries but I never really used them. Since each fontface contained contained about 450 glyphs, that means that my program now has to store 900 unique paths less after initial font loading. So far it has been running continously for almost 2 hours and not a single crash occurred - it's something that never happened before.

Since I have your attention, I would like to ask somewhat related question.
This is the output of vcdbg reloc stats when my program has been running for 2 hours:

pi@raspberrypi ~ $ sudo vcdbg reloc stats

Relocatable heap version 4 found at 0x1c000000
total space allocated is 44M, with 42M relocatable, 2.3M legacy and 0 offline
1 legacy blocks of size 2359296

small_allocs : 1655725
allocs : 1498127
alloc_fails : 12
legacy_block_fails : 0
compactions : 1
discard_compactions : 0
aggressive_compactions : 0
aggressive_compaction_waits : 0
aggressive_compaction_timeouts: 0
locks : 0
small_locks : 0

Thing is, the bold numbers keep on increasing. I assume they count show TOTAL memory allocations count, not CURRENT memory allocations count and it means nothing that I shouldn't be worried about it?

This part hasn't really changed at all:
free list at 0x1dc1c740
3.9M free memory in 6 free block(s)
largest free block is 3.8M bytes

Guess that based on this, I can exclude any memory leak?

[pawelduda]

I have left the program running overnight. So far 15 hours without crash, the problem seems fixed. I will commit changes I've made to my repository later on.

[pawelduda]

In case someone runs into the same issue, I have managed to find out the exact cause. It's all described here:
https://github.com/ajstarks/openvg/issues/22

[BorreF]

I've also experienced GPU memory corruption and having debugged this for quite some time now I'm leaning towards a video core firmware issue (it seems to be more frequent when running on the RPi2 model).

Symptom is that font glyphs become corrupted (after running for hours/days) and finally a function will fail with the VG_OUT_OF_MEMORY error. I've pretty much ruled out HW since it is the same glyph/character that gets corrupted (provided no other conditions change). If I change the GPU memory split or shuffle my code a bit (such as using vgClearPath as opposed to vgCreatePath/vgDestroyPath) I can see different glyphs corrupted, but it's still repeatable. Time however is not repeatable as it could range anywhere from minutes to several days.

quote] I believe that calling functions like Rect() or Circle() multiple in a loop caused this behavior. This is due to the fact that those functions create and then destroy a new path each time they're called, meaning that a lot of memory reallocation is going on and it's likely that other paths existing in memory could have their segments data overwritten in process (?)[/quote]

The above quote is a possible explanation, but there is nothing wrong with creating and destroying paths and in no case should this lead to corruption of GPU memory (it may lead to fragmentation and out-of-memory issues, but not corruption).

There appears to be no single call or sequence of calls that lead to corruption so possibly it is timing related (such as a garbage collector conflicting with the running application).

I'm currently at a dead-end and thinking there is something fundamentally "buggy" about the RPi graphics implementation that will need to be sorted out by the Raspberry folks.

[pawelduda]

Hey BorreF, I'm glad someone else can relate to this issue I had.

I have refactored my code so that it does not reallocate memory for paths and what not - since then 2 months have been passed (that's hundreds of hours in total of running my program) and I haven't seen the problem come back for even once. Every single path, paint or image is being created outside of the main loop.

Though when it comes to the memory reallocation not being the root of the problem, I can totally agree with you. It should slow down the program in the worst case but definitely not corrupt the other parts of memory. I have been developing a Raspberry Pi powered digital signage program as a part of my thesis project and have wasted around a month because of this issue. Before C OpenVG implementation I tried node.js openvg-canvas module (which basically renders fullscreen hardware accelerated html5 canvas). As far as I remember, there was the exact same issue with either glyphs being corrupted or paths flashing randomly which ultimately made me switch to C implementation.

Shame that I wasn't aware of the cause back then. Having minimal experience in OpenVG, I became a victim of tricking myself into thinking that the cause were my bad coding practices and kept making changes to my own code thinking it would fix that. Well, it eventually did but totally by an accident.

[SteveA]

I have the same or similar problems to those as described here. I use RP2B with the latest OS and firmware updates and the latest openvg-master (as at 2-9-15). After about 90 mins, the screen gets currupted, fonts are drawn incorrectly and in addition, when I quit my application (with the necessary finish() as described in the documentation) the desktop size has reduced by a slight amount. I have to reboot to get everything back to normal.

If I run my program for say 10 mins then quit, then restart then quit again after 10 mins, and so on all day, all is well. However if I leave the program running (which is just a graphics display loop consisting of rect/line/prompts drawn on the screen) it eventually corrupts.

I saw the post by paweldu Feb 21 which helpfully points to the cause of the issues at https://github.com/ajstarks/openvg/issues/22 and I see that the latest version of openvg-master has got all of the fixes done and that post has been closed, however, my fault still persists and I do not know how to cure it. I am using the latest versions and it is only when it is left running for a long time that a fault develops and nothing in the loop changes.

If anyone has found an actual fix I could really do with some help because my application has to run all day without any corruption.

Thanks for looking
Steve

[pawelduda]

I have the same or similar problems to those as described here. I use RP2B with the latest OS and firmware updates and the latest openvg-master (as at 2-9-15). After about 90 mins, the screen gets currupted, fonts are drawn incorrectly and in addition, when I quit my application (with the necessary finish() as described in the documentation) the desktop size has reduced by a slight amount. I have to reboot to get everything back to normal.

If I run my program for say 10 mins then quit, then restart then quit again after 10 mins, and so on all day, all is well. However if I leave the program running (which is just a graphics display loop consisting of rect/line/prompts drawn on the screen) it eventually corrupts.

I saw the post by paweldu Feb 21 which helpfully points to the cause of the issues at https://github.com/ajstarks/openvg/issues/22 and I see that the latest version of openvg-master has got all of the fixes done and that post has been closed, however, my fault still persists and I do not know how to cure it. I am using the latest versions and it is only when it is left running for a long time that a fault develops and nothing in the loop changes.

If anyone has found an actual fix I could really do with some help because my application has to run all day without any corruption.

Thanks for looking
Steve

Could you post the part of code inside the loop? As far as I remember, the issue was closed but I'm not sure if API update has been released to address it. After avoiding code that caused this to happen my software has been running flawlessly for months of uptime.

[SteveA]

Thanks for letting me know that there could have been an API update. I have just downloaded the whole openvg-master folder again and and have noticed that, for example, the shapes.c file creation date is later than my previous shapes.c file, although on https://github.com/ajstarks/openvg the date comments do not not show a change of this file for the past 7 months. I am hoping that this cures the problem. I will recompile, run (overnight if possible) and report back.

Thanks
Steve

[SteveA]

Thanks again for the heads up. I downloaded the openvg afresh, recompiled and it has run fine for over 8 hours without any corruption (I am continuously looping and displaying 24-bit ADC data onto a panel in a large font size). I am much relieved. I cannot see a change of date from when I last downloaded openvg-master from the github site, however the fresh download has cured it.

Out of interest only:
I would like to use a different mono-spaced font to use one without a dot in the 0 and have tried to follow the instructions on how to create a new font inc file. My ineptitude and lack of understanding has not got me very far. Does anyone know if there is a simplified step-by-step note on how to create a new font file.

Also, the SaveEnd() function saves the raster image but as yet I have not had any joy displaying the picture file on a PC. Has anyone done this and if so, what PC application was used?

Thanks again
Steve

[ajstarks]

@SteveA: OpenVG library author here: I'm glad the library is working for you.

Re: SaveEnd() -- the program https://github.com/ajstarks/openvg/blob ... raw2png.go will convert this raw file to png. (If you don't want to install Go, I can probably make a binary for your system)

Re: fonts. What is the exact problem you are having?

[SteveA]

Hello ajstarks

You've done a really excellent job with your openvg and my program is now looping away merrily, displaying oscilloscope displays, etc. I am really impressed with your routines because I can now display graphical data on the Raspberry Pi with clear text, and not just display scrolling data in a terminal window.

About SaveEnd()
I wanted to grab a screen shot of the oscillograph displays and save it onto my small NAS network to be displayed on a remote computer, possibly a PC. The screen shots are being saved OK but I could not locate a program to display the raw graphics on a PC, hence my query. I didn't want to install Go as I wanted the display to be done on a remote PC/Tablet seemlessly and not necessarily be displayed on the Raspberry Pi.

On my wish list, it would be nice to be able to change the existing routine to be called such as SaveRAWEnd(), and add other routines into your project called SavePNGEnd(), SaveJPGEnd(), so I could change the type of picture format being saved.

About fonts
It is purely my understanding of the individual steps on how to get a font, which looks fine and so ought to be used, then converting it to glyphs then an inc file. This problem is down to me and so I need to go through it again to get the steps correct.

Using const char * for strings
May I ask if there is a way to change your lib to also accept const char * strings (instead of or as well as standard char[] array strings) without the compiler complaining which occurs when you have all warnings enabled). I only ask because when using multi-dimension prompts/strings, I need to use multi-dim arrays of pointers to the actual string, because they occupy a lot less space than a multi-dim array of chars (because, as you will know, the latter needs to allocate space for the largest string throughout the array and so wastes space.) When using a massive number of multi-dimensional prompt arrays it can get wasteful and memory starts being eaten away.

Once again, thank you for an excellent library, which has brought graphics facilities to such as me, a relative beginner and struggler with it all as my experience todate has not been on Linux, but small embedded micros. It is a learning curve.
Thanks
Steve

[SteveA]

Further to the above I still have a problem where the graphics window gets corrupted after a long time, about 45 mins of continuous looping, which has only come to light this weekend due to the number and frequency of prompts being written to the screen.

I am using Raspberry Pi 2, not overclocked, with latest updates of the OS, firmware and openvg-master (as of 2 days ago.)
I use 256MB for the GPU in the memory split setting.

I read an ADC, calculate values and display the results in a graphics window in different sized and colour fonts. I also display the ADC as an oscilloscope display and loop continuously until a key is pressed. It will happily run for about 45mins then several things may happen:

1. The background is normally red and the text (28 point) is white. After about 45 mins, one or more triangular shapes will draw themselves in white on the screen and the looping will gradually slow down. I can still exit on a key press and rerun the program again OK until it happens once more after some time.

or

2. The whole system will stop looping (or become so slow as to give the appearance it has stopped) and the only way to get out is to force a break by pressing the Pause/break key.

However, if 2) has happened, when I now try to rerun the program it will not run as in 1) - instead it will hang at the point when calling the init(..) routine (which is in shapes.h). To check, I put a printf() prompt before and after the call to init(..) and do not get the after prompt so it appears that it has not returned from init(..).

The program will now not rerun at all.

However, if I reboot the RPI and rerun the program (without any changes at all) all is fine and it will once more run again for about 45 mins until 1) or 2) happens.

It is as though memory is being created, hogged and not released correctly (and a glyph gets drawn across the screen as in 1). I am not creating or releasing memory blocks in my code - all memory is fixed global arrays.

For info: I am using an additional font (digit7) and use it as described by pearyn in another post. I write 3 digits in that font in another window and use the standard built in fonts (Serif, etc) to display the looping ADC results

Can anyone suggest how to track this problem down and how to monitor memory usage to see what is being used and what is not released.

I don't have another problem with the openvg that I know of, but this could be because in my other projects I only write a few graphic text prompts every so often, whereas this project is continously looping and updating the graphics text prompts every 100mS and so if there is an any memory problem it would be seen quicker.

Update: when I look at memory usage from the command line using:
$ sudo vcdbg reloc stats

1. Before running the program:

Relocatable heap version 4 found at 0x2f000000
total space allocated is 236M, with 236M relocatable, 0 legacy and 0 offline
0 legacy blocks of size 2359296

small_allocs : 23
allocs : 13
alloc_fails : 0
legacy_block_fails : 0
compactions : 0
discard_compactions : 0
aggressive_compactions : 0
aggressive_compaction_waits : 0
aggressive_compaction_timeouts: 0
locks : 0
small_locks : 0

free list at 0x3d897920
233M free memory in 1 free block(s)
largest free block is 233M bytes

2. After running for awhile and seeing the looping slowing down, I exit from the program and this is typical of what I get:

Relocatable heap version 4 found at 0x2f000000
total space allocated is 236M, with 234M relocatable, 0 legacy and 2.3M offline
0 legacy blocks of size 2359296

small_allocs : 2019
allocs : 5827729
alloc_fails : 0
legacy_block_fails : 0
compactions : 0
discard_compactions : 0
aggressive_compactions : 0
aggressive_compaction_waits : 0
aggressive_compaction_timeouts: 0
locks : 0
small_locks : 0

free list at 0x3d796860
invalid space 0x1465210709 in free list at 0x3c1c8b00
1.1M free memory in 9 free block(s)
largest free block is 438K bytes

I am not sure what this means, but it seems as though the largest free block has dropped down from 233MB to 438KB, with 5827729 allocs.

As mentioned, when I then try to re-run the program, it cannot get passed the graphics init(...). Is there any method in software to forceably clear out the memory. The only way I know is to reboot the Raspberry Pi! Any pointers would be gratefully received.


Thanks for looking
Steve

[ajstarks]

Can you try running your program without any text display?
I'm wondering if that is where the problem is.

Also, is the code available so that I can reproduce the bug?

[SteveA]

Hello ajstarks,

I have cut sections out bit by bit and have done a test program of my project without my isolated ADC and input/output boards fitted. Could you please pm me to let me know where to email it. Thanks