Strange segfault after several hours of running program

[ktb92677]

Hi all, I created a program fairly closely related to hello_pi/hello_video.c. And for some reason after several hours of running the program I get these very strange segfaults that I cannot for the life of me figure out how to resolve. Here is what I am running:

- Raspberry PI 3B+
- Raspbian Lite (Buster)

I have checked the following:

- I am not under voltage. I am using a 5.1v 2.5a power supply from raspberry PI themselves.
- I did not over clock the system. I am sitting at 1.4 like a normal PI.
- The system has heat sinks and a fan to keep the temperature low.
- The system is also totally up to date (apt update... upgrade... rpi-update etc)

Here are a few of the errors I am receiving from the address sanitizer built into gcc:

Code: Select all

==26987==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x45009021 at pc 0x769ca548 bp 0x5a7fa474 sp 0x5a7fa040
READ of size 8 at 0x45009021 thread T450 (ILCS_HOST)
    #0 0x769ca547  (/usr/lib/arm-linux-gnueabihf/libasan.so.5+0x3a547)

Address 0x45009021 is a wild pointer.
SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib/arm-linux-gnueabihf/libasan.so.5+0x3a547) 
Shadow bytes around the buggy address:
  0x28a011b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a011c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a011d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a011e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a011f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x28a01200: fa fa fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa
  0x28a01210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a01220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a01230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a01240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x28a01250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
Thread T450 (ILCS_HOST) created by T443 here:
    #0 0x769db9c7 in pthread_create (/usr/lib/arm-linux-gnueabihf/libasan.so.5+0x4b9c7)
    #1 0x7693b203 in vcos_thread_create /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:212

Thread T443 created by T0 here:
    #0 0x769db9c7 in pthread_create (/usr/lib/arm-linux-gnueabihf/libasan.so.5+0x4b9c7)
    #1 0x74ee1c57 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/arm-linux-gnueabihf/libstdc++.so.6+0x9dc57)
    #2 0x6bb02a7f  (<unknown module>)

==26987==ABORTING

Code: Select all

==23420==ERROR: AddressSanitizer: SEGV on unknown address 0x0019c2c0 (pc 0x74b060f0 bp 0x74b08f3c sp 0x6acfe3b8 T2)
==23420==The signal is caused by a READ memory access.
    #0 0x74b060ef in completion_thread (/opt/vc/lib/libvchiq_arm.so+0x20ef)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/vc/lib/libvchiq_arm.so+0x20ef) in completion_thread
Thread T2 (VCHIQ completio) created by T0 here:
    #0 0x769609c7 in pthread_create (/usr/lib/arm-linux-gnueabihf/libasan.so.5+0x4b9c7)
    #1 0x768c0203 in vcos_thread_create /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:212

==23420==ABORTING

Here is the output from gdb during those crashes:

Code: Select all

Thread 3 "VCHIQ completio" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x6fbfeb00 (LWP 15664)]
0x751c60f0 in completion_thread () from /opt/vc/lib/libvchiq_arm.so

All the crashes seem to be related to "completion_thread()". I have no idea what that is.

Any help would be greatly appreciated!

[jdonald]

Saw your post on Stack Exchange that led me here.

I think joan's comment is the most insightful so far... showing that this problem is common with hello_video.c would confirm it's not caused by your code.

However, this exact crash certainly wouldn't appear in the vanilla examples because people don't normally build the hello_pi examples with address sanitizer. And that could be why such issues can linger in these libraries for a long time. Note this is an invalid read, not an invalid write, so production code can get away with it being benign a good portion of the time especially if it usually reads out zero.

It looks like you've got a production setup that needs to run for days, so it sounds important to fix this even if it takes hours to reproduce and debug. If I were in your situation I would suggest narrowing down the offending read further. The source code for completion_thread() in libvchiq_arm.so can be found here: https://github.com/raspberrypi/userland ... chiq_lib.c and perhaps you could build that with debug symbols and asan enabled.

Hypothetically if the bug turned out to be there in the VideoCore libraries and you proposed a proper fix, you wouldn't really need to wait on your fix getting accepted upstream. You could just run with your own custom build of livchiq_arm.so. Unless the root of the problem was kernel/firmware-side, but at this point it looks most likely that the bug is in userland, either your code or the library.

Hope that helps point you in the next direction.

[ktb92677]

Do you know at all under what circumstances "completion_thread()" is called?

[jdonald]

Reading the code, it looks like it spawns that thread early on in vchiq_connect() then it's basically a waiting loop until it's time to shutdown all resources. I'm no expert, just looking at this for the first time like you.

Interestingly if you do a search for completion_thread() in the repo's issues the one that pops up is this one: https://github.com/raspberrypi/userland/issues/533 where a user got a crash in that area due to his double-deletion. While most of such errors in a user's program get caught by address sanitizer, asan can't immediately point out a problem of just using the API incorrectly.

Also I wonder if you'd get more constructive feedback if you posted this as a GitHub issue on there. The gang on Stack Exchange are all trying to help but they have different areas of expertise. User @6by9 is here on the forums too so you should link this thread. Only downside of a GitHub issue is that normally those should be reserved for bugs in raspberrypi/userland, not bugs in your code, but at this point we don't know which it is.

By the way, I'm assuming those suggestions of running a watchdog and restarting after several hours aren't acceptable? Like you have a production application that isn't ideal for periodic interruptions.

Also btw, I realized my comment about the read being benign in other programs was incorrect. The segmentation fault is due to the read being outside the program's address space, so it would crash in production regardless of whether you've built with asan.

[ktb92677]

I just read through the link you posted. Looks like there's also a program called KODI that also has some segfaults related to this mysterious "completion_thread()". But nothing really constructive.

I have just posted this issue to the github as well. How can I link @6by9 in this thread?

Yes this is a production application that can't just be randomly crashing.

[6by9]

VCHIQ is the RPC between the ARM and the VideoCore VPU.
The completition thread handles the callbacks from the VPU informing the ARM that something has happened. The completition thread then passes this off to the relevant VCHIQ client (most likely ilcs and ilclient in this case).

Sorry, without full code there is pretty much nothing I can do. As asked above, can you provoke this with hello_video without modifications?

What is your application actually trying to do?
We are trying to move away from IL as it is a dead API (all participants on the working group dropped out in around 2014). It's also a total swine to work with as it just sulks if you don't poke it in exactly the right way. I suspect there is a better solution to your problem than fighting IL.

[ktb92677]

So on getting hello_video to have the same exact issue I am seeing here I am not sure. As of this minute I have a begun a test now but it will take at least a full day for me to know for sure.

I understand without the code base it will be difficult to debug this issue... but considering the time investment it takes for the issue to actually pop up I was hoping to narrow down at least if its my codes fault or the backend... It also seems like gdb and address sanitization will be useless in my attempts at actually solving this problem considering neither even mentions anything related to my code base.

My actual application is a live streaming. I am being delivered raw h264 packets and in a hardware accelerated fashion I want to decode them to the screen. I believe I am using ilclient. Is there any better way to do it? Low latency is extremely key.

[ktb92677]

Okay small update. Here's the backtrace that is showing on gdb when that crash happens:

Code: Select all

Thread 3 "VCHIQ completio" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x6fbfeb00 (LWP 23464)]
0x751c60f0 in completion_thread () from /opt/vc/lib/libvchiq_arm.so
(gdb) backtrace
#0  0x751c60f0 in completion_thread () at /opt/vc/lib/libvchiq_arm.so
#1  0x76f68cb0 in vcos_thread_entry (arg=0x751d9318 <vchiq_instance+16>)
    at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#2  0x7539f494 in start_thread (arg=0x6fbfeb00) at pthread_create.c:486
#3  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
(gdb) 

I am going crazy here trying to solve this. 6by9 please help!

[ktb92677]

I also have the output of "thread apply all bt"

Code: Select all

(gdb) thread apply all bt
 
Thread 206345 (Thread 0x6b5fdb00 (LWP 8906)):
#0  0x7533040c in __lll_lock_wait_private (futex=futex@entry=0x753957d4 <main_arena>) at ./lowlevellock.c:33
#1  0x752b9ec8 in reused_arena (avoid_arena=0x0) at arena.c:839
#2  0x752b9ec8 in arena_get2 (size=320, avoid_arena=0x0) at arena.c:916
#3  0x752bd2f0 in tcache_init () at malloc.c:2978
#4  0x752be960 in tcache_init () at malloc.c:3113
#5  0x752be960 in __GI___libc_free (mem=0xd9d68) at malloc.c:3113
#6  0x0002d598 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(AUDIO_CLIENT*), AUDIO_CLIENT*> > >::~_State_impl() (this=0xd9d68, __in_chrg=<optimized out>) at /usr/include/c++/8/thread:188
#7  0x7550f9c0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#8  0x7539f494 in start_thread (arg=0x6b5fdb00) at pthread_create.c:486
#9  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 206335 (Thread 0x6c9f9b00 (LWP 8885)):
#0  0x752ea788 in arch_fork (ctid=0x6c9f9b68) at ../sysdeps/unix/sysv/linux/arch-fork.h:40
#1  0x752ea788 in __libc_fork () at ../sysdeps/nptl/fork.c:76
#2  0x752a9238 in _IO_new_proc_open
    (fp=fp@entry=0x6be27568, command=command@entry=0x6acc8528 "echo -n $(find /sys/class/net -follow -maxdepth 2 -name wireless 2>/dev/null | cut -d'/' -f5 | sed -n 1p) 2>&1", mode=<optimized out>, mode@entry=0x76eae51c "r") at iopopen.c:122
#3  0x752a9500 in _IO_new_popen
    (command=0x6acc8528 "echo -n $(find /sys/class/net -follow -maxdepth 2 -name wireless 2>/dev/null | cut -d'/' -f5 | sed ---Type <RET> for more, q to quit, c to continue without paging--c
n 1p) 2>&1", mode=0x76eae51c "r") at iopopen.c:203                                                                          
#4  0x76ea324c in system_output(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () at /usr/local/lib/libUtil.so                                                                                                          
#5  0x00040338 in System_WIFI_Network_Interface[abi:cxx11]() () at /usr/include/c++/8/bits/char_traits.h:287                
#6  0x00040338 in System_Has_ETHERNET() () at screens/wifi/Wifi_Utility.cpp:241                                              
#7  0x00047290 in Check_Wifi_Battery(UTILITY_CORNER*) (utility_corner=0x9c700) at screens/corners/Utility_Corner.cpp:474    
#8  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6                                                        
#9  0x7539f494 in start_thread (arg=0x6c9f9b00) at pthread_create.c:486                                                      
#10 0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73                                                          
                                                                                                                             
Thread 106 (Thread 0x6d1fab00 (LWP 24539)):                                                                                  
#0  0x753a9088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x234b58) at ../sysdeps/unix/sysv/linux/futex-internal.h:205                                                                                            
#1  0x753a9088 in do_futex_wait (sem=sem@entry=0x234b58, abstime=0x0) at sem_waitcommon.c:115                                
#2  0x753a91f4 in __new_sem_wait_slow (sem=0x234b58, abstime=0x0) at sem_waitcommon.c:282                                    
#3  0x758a19c8 in mmal_component_action_thread_func () at /opt/vc/lib/libmmal_core.so                                        
#4  0x76f68cb0 in vcos_thread_entry (arg=0x234a70) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144                                                                                                                    
#5  0x7539f494 in start_thread (arg=0x6d1fab00) at pthread_create.c:486                                                      
#6  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73                                                          
                                                                                                                             
Thread 64 (Thread 0x622fab00 (LWP 24489)):                                                                                  
#0  0x753a9088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x6be1e394) at ../sysdeps/unix/sysv/linux/futex-internal.h:205                                                                                          
#1  0x753a9088 in do_futex_wait (sem=sem@entry=0x6be1e394, abstime=0x0) at sem_waitcommon.c:115                              
#2  0x753a91f4 in __new_sem_wait_slow (sem=0x6be1e394, abstime=0x0) at sem_waitcommon.c:282                                  
#3  0x76faa8bc in ilcs_execute_function_ex () at /opt/vc/lib/libopenmaxil.so                                                
#4  0x76fab8a4 in ilcs_pass_buffer () at /opt/vc/lib/libopenmaxil.so                                                        
#5  0x0002da70 in Render_H264_Frame(OPENMAX_H264_DECODER*, char*, int) (decoder=0x6be13a08, buffer=0x20901c "", size=1370) at openmax/OPENMAX_H264_DECODER.cpp:114
#6  0x000327d8 in Receive_Thread(LIVE_444_CLIENT*) (live=0x92be0) at live444/LIVE_444_CLIENT.cpp:27
#7  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#8  0x7539f494 in start_thread (arg=0x622fab00) at pthread_create.c:486
#9  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 63 (Thread 0x62afbb00 (LWP 24488)):
#0  0x753a9e98 in __lll_lock_wait (futex=futex@entry=0x751d92ec <vchiq_lib_mutex>, private=0) at lowlevellock.c:46
#1  0x753a1f44 in __GI___pthread_mutex_lock (mutex=0x751d92ec <vchiq_lib_mutex>) at pthread_mutex_lock.c:80
#2  0x751c6d18 in vchiq_release_message () at /opt/vc/lib/libvchiq_arm.so
#3  0x76faaff4 in ilcs_task () at /opt/vc/lib/libopenmaxil.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x6be1e1c8) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x7539f494 in start_thread (arg=0x62afbb00) at pthread_create.c:486
#6  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 62 (Thread 0x632fcb00 (LWP 24487)):
#0  0x753a61a0 in futex_wait_cancelable (private=0, expected=0, futex_word=0x6be1e1a4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0x753a61a0 in __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x6be1e178) at pthread_cond_wait.c:502
#2  0x753a61a0 in __pthread_cond_wait (cond=cond@entry=0x6be1e178, mutex=0x0, mutex@entry=0x6be1e15c) at pthread_cond_wait.c:655
#3  0x76f68d94 in _timer_thread (arg=0x6be1e158) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:722
#4  0x7539f494 in start_thread (arg=0x632fcb00) at pthread_create.c:486
#5  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 59 (Thread 0x6bdfeb00 (LWP 24484)):
#0  0x753a0a3c in __GI___pthread_timedjoin_ex (threadid=1801444096, thread_return=0x0, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:89
#1  0x7550fbf4 in std::thread::join() () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#2  0x0002baa4 in Audio_Client_Loop(AUDIO_CLIENT*) (client=0x6be09e00) at audio/AUDIO_CLIENT.cpp:227
#3  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x7539f494 in start_thread (arg=0x6bdfeb00) at pthread_create.c:486
#5  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 58 (Thread 0x6d9fbb00 (LWP 24483)):
#0  0x753a36e4 in __pthread_mutex_unlock_usercnt (mutex=<optimized out>, decr=1) at pthread_mutex_unlock.c:56
#1  0x76ea6048 in Send_Keyboard_Data(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#2  0x76ea6c38 in Send_Thread(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#3  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x7539f494 in start_thread (arg=0x6d9fbb00) at pthread_create.c:486
#5  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 50 (Thread 0x70508b00 (LWP 24339)):
#0  0x7531b6ac in __GI___select (timeout=0x70508418, exceptfds=0x0, writefds=0x0, readfds=0x70508458, nfds=11) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x7531b6ac in __GI___select (nfds=11, readfds=0x70508458, writefds=0x0, exceptfds=0x0, timeout=0x705083e0) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x757dcaec in Listen_Keyboard(KEYBOARD*) () at /usr/local/lib/libgui.so
#3  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x7539f494 in start_thread (arg=0x70508b00) at pthread_create.c:486
#5  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 16 (Thread 0x642feb00 (LWP 23535)):
#0  0x75317cd0 in __GI___poll (timeout=1, nfds=1, fds=0x6fce4380) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x75317cd0 in __GI___poll (fds=0x6fce4380, nfds=1, timeout=1) at ../sysdeps/unix/sysv/linux/poll.c:26
#2  0x757df3cc in Listen_Mouse_Once(MOUSE*) () at /usr/local/lib/libgui.so
#3  0x76eaa86c in Read_Mouse_Data(DEVICE_NODE*) () at /usr/local/lib/libUtil.so
#4  0x76eacd18 in Read_Thread(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#5  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#6  0x7539f494 in start_thread (arg=0x642feb00) at pthread_create.c:486
#7  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 15 (Thread 0x66ffeb00 (LWP 23534)):
#0  0x7531b6ac in __GI___select (timeout=0x66ff63f8, exceptfds=0x0, writefds=0x0, readfds=0x66ff6458, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x7531b6ac in __GI___select (nfds=5, readfds=0x66ff6458, writefds=0x0, exceptfds=0x0, timeout=0x66ff63f0) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x76ea93ac in Refresh_Thread_DEVICE_MANAGER(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#3  0x7550f9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x7539f494 in start_thread (arg=0x66ffeb00) at pthread_create.c:486
#5  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 6 (Thread 0x6e1fcb00 (LWP 23467)):
#0  0x753a9088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa665c <cecservice_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753a9088 in do_futex_wait (sem=sem@entry=0x76fa665c <cecservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753a91f4 in __new_sem_wait_slow (sem=0x76fa665c <cecservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f8ce98 in cecservice_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa6670 <cecservice_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x7539f494 in start_thread (arg=0x6e1fcb00) at pthread_create.c:486
#6  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 5 (Thread 0x6e9fdb00 (LWP 23466)):
#0  0x753a9088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa58cc <tvservice_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753a9088 in do_futex_wait (sem=sem@entry=0x76fa58cc <tvservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753a91f4 in __new_sem_wait_slow (sem=0x76fa58cc <tvservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f8bb28 in tvservice_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa58e0 <tvservice_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x7539f494 in start_thread (arg=0x6e9fdb00) at pthread_create.c:486
#6  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 4 (Thread 0x6f1feb00 (LWP 23465)):
#0  0x753a9088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa6758 <dispmanx_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753a9088 in do_futex_wait (sem=sem@entry=0x76fa6758 <dispmanx_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753a91f4 in __new_sem_wait_slow (sem=0x76fa6758 <dispmanx_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f903a4 in dispmanx_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa7498 <dispmanx_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x7539f494 in start_thread (arg=0x6f1feb00) at pthread_create.c:486
 
Thread 3 (Thread 0x6fbfeb00 (LWP 23464)):
#0  0x751c60f0 in completion_thread () at /opt/vc/lib/libvchiq_arm.so
#1  0x76f68cb0 in vcos_thread_entry (arg=0x751d9318 <vchiq_instance+16>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#2  0x7539f494 in start_thread (arg=0x6fbfeb00) at pthread_create.c:486
#3  0x75322578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
 
Thread 1 (Thread 0x7050c010 (LWP 23459)):
#0  0x7531a51c in ioctl () at ../sysdeps/unix/syscall-template.S:78
#1  0x751c6c08 in vchiq_queue_message () at /opt/vc/lib/libvchiq_arm.so
#2  0x75861fe0 in mmal_vc_send_message () at /opt/vc/lib/libmmal_vc_client.so
#3  0x7586546c in mmal_vc_port_send () at /opt/vc/lib/libmmal_vc_client.so
#4  0x7589dff0 in mmal_port_send_buffer () at /opt/vc/lib/libmmal_core.so
#5  0x0001b254 in Render_Buffer(RANA_EXT*, MMAL*) (render_context=0x7efff250, rana_ext=0x7efff270) at main.cpp:53
#6  0x0001b254 in main() () at main.cpp:398

6by9 please help!

[ktb92677]

VCHIQ is the RPC between the ARM and the VideoCore VPU.
The completition thread handles the callbacks from the VPU informing the ARM that something has happened. The completition thread then passes this off to the relevant VCHIQ client (most likely ilcs and ilclient in this case).

Sorry, without full code there is pretty much nothing I can do. As asked above, can you provoke this with hello_video without modifications?

What is your application actually trying to do?
We are trying to move away from IL as it is a dead API (all participants on the working group dropped out in around 2014). It's also a total swine to work with as it just sulks if you don't poke it in exactly the right way. I suspect there is a better solution to your problem than fighting IL.

6by9 can you take a look at my replies?

[jdonald]

Some thoughts:
* Let's cross-link the GitHub ticket while we're at it: https://github.com/raspberrypi/userland/issues/563
* And the Stack Exchange one: https://raspberrypi.stackexchange.com/q ... ng-program
* You may want to use the 'Edit' button rather than follow-on posting four times! Forum etiquette standards on this are unclear, but regardless it can be helpful for you to present everything coherently.
* I stumbled upon your other post on GitHub and this forum from April. Relating to that, by chance does your program already incorporate the sample hack that 6by9 made for you here? https://github.com/6by9/userland/commit ... 209fbe1f12 If so, well that's a hack and all bets are off as to whether it could stream for hours.
* From the gdb stack trace it looks like more symbols are already in there than I thought so I guess no need to recompile libvchiq_arm.so in debug. I'm not sure why the asan stack traces didn't show as much detail earlier.
* Not sure if you are running in gdb live--which would slow things down. For something that takes hours to crash I recommend instead doing ulimit -c unlimited then examining the core dump after the fact.
* For further debugging of this stack frame you can print out specific variables in completion_thread() using p
* You had answered the Stack Exchange gang that you're nowhere near running out of memory, but then I thought what about gpu_mem? Any change in how long it takes to crash if you reduce or increase that in config.txt?

[ktb92677]

Some thoughts:
* Let's cross-link the GitHub ticket while we're at it: https://github.com/raspberrypi/userland/issues/563
* And the Stack Exchange one: https://raspberrypi.stackexchange.com/q ... ng-program
* You may want to use the 'Edit' button rather than follow-on posting four times! Forum etiquette standards on this are unclear, but regardless it can be helpful for you to present everything coherently.
* I stumbled upon your other post on GitHub and this forum from April. Relating to that, by chance does your program already incorporate the sample hack that 6by9 made for you here? https://github.com/6by9/userland/commit ... 209fbe1f12 If so, well that's a hack and all bets are off as to whether it could stream for hours.
* From the gdb stack trace it looks like more symbols are already in there than I thought so I guess no need to recompile libvchiq_arm.so in debug. I'm not sure why the asan stack traces didn't show as much detail earlier.
* Not sure if you are running in gdb live--which would slow things down. For something that takes hours to crash I recommend instead doing ulimit -c unlimited then examining the core dump after the fact.
* For further debugging of this stack frame you can print out specific variables in completion_thread() using p
* You had answered the Stack Exchange gang that you're nowhere near running out of memory, but then I thought what about gpu_mem? Any change in how long it takes to crash if you reduce or increase that in config.txt?

Firstly thank you so much for being so active with me in trying to solve this issue. It is so greatly appreciated!

Yes 6by9 created that hack for me and I am currently using that. Here is my current source code for the h264 decoder (based on 6by9's code). https://pastebin.com/6v9najNs. But again we can't even be 100% sure that the issue even lies in that code.

Yes I am running gdb live, and it is fairly slow unfortunately... I will try getting a coredump as well now but I can't guarantee when i can post it.

On GPU memory that is an interesting question... let me see if it happens any faster with less memory. I will report back on that

[ktb92677]

Tested the memory, for some reason no number below 128 works. There don't seem to be any errors just a black screen. Is there any took I can use to watch GPU memory usage for the raspberry PI?

[jdonald]

I have not tried this myself, but according to this topic you can get stats on the GPU heap via vcgencmd. Hopefully that works both in legacy and fkms mode.

[ktb92677]

Okay I think I just made some fairly significant progress. I am using both ilclient and openvg at the same time in my program. I decided to replace openvg with a simple /dev/fb0 implementation to do my rendering (I render objects along side the h264 stream). After this the system does not segfault!!!! The problem might lie in openvg. As I was looking through openvg's bug history I noticed the following issue: https://github.com/ajstarks/openvg/issues/68. This is oddly reminiscent of the issue I am experiencing. I would assume omxplayer also uses ilclient. It would seem that ilclient and openvg do not play well together. I am drafting a massive post for them right now as we speak. My concern is that library hasn't had an update in almost a year. This is a bit concerning.

As a side note if I were to replace openvg what could I use? /dev/fb0 + MMAL are far too slow for what I want. OPENGLES2 is far too complex for my needs (don't need any 3D and definitely don't want to learn how to use shaders). I just need a nice well maintained hardware accelerated 2D graphics renderer that will work without and dependencies on X11.

[6by9]

Looking at the backtraces you're mixing MMAL and IL? Why? What is the overall intent of this application?

They should co-exist, but I can't say for definite that there aren't interactions.

And now you're saying you're using VG too? Wowsers, far too many variables to try and track down any lurking issue.

edit: "sudo vcdbg reloc" is the best way to get gpu heap stats. "vcgencmd get_mem gpu" will only tell you how much has been assigned to the gpu.

Further edit: be aware that OpenVG is not directly supported on the Pi4. It uses the 3D hardware, and there are no firmware drivers for that. At one point it was supported by Mesa, but no longer. https://www.raspberrypi.org/forums/view ... 3&t=245380

[ktb92677]

Looking at the backtraces you're mixing MMAL and IL? Why? What is the overall intent of this application?

They should co-exist, but I can't say for definite that there aren't interactions.

And now you're saying you're using VG too? Wowsers, far too many variables to try and track down any lurking issue.

edit: "sudo vcdbg reloc" is the best way to get gpu heap stats. "vcgencmd get_mem gpu" will only tell you how much has been assigned to the gpu.

Further edit: be aware that OpenVG is not directly supported on the Pi4. It uses the 3D hardware, and there are no firmware drivers for that. At one point it was supported by Mesa, but no longer. https://www.raspberrypi.org/forums/view ... 3&t=245380

The overall intent of the application is a remote desktop application. So I want to play an h264 stream and I want to be able to render UI elements ontop of it. (mouse, settings button etc).

Do you have any suggestions on a good replacement for openvg? I want a hardware accerated 2D renderer that is not dependent on X11 and also does not have me learning how to use shaders. Thanks!

[6by9]

The overall intent of the application is a remote desktop application. So I want to play an h264 stream and I want to be able to render UI elements ontop of it. (mouse, settings button etc).

But why the mishmash of IL and MMAL?

Do you have any suggestions on a good replacement for openvg? I want a hardware accerated 2D renderer that is not dependent on X11 and also does not have me learning how to use shaders. Thanks!

Read the other thread. There are implementations of OpenVG on top of GL.

[ktb92677]

The overall intent of the application is a remote desktop application. So I want to play an h264 stream and I want to be able to render UI elements ontop of it. (mouse, settings button etc).

But why the mishmash of IL and MMAL?

Do you have any suggestions on a good replacement for openvg? I want a hardware accerated 2D renderer that is not dependent on X11 and also does not have me learning how to use shaders. Thanks!

Read the other thread. There are implementations of OpenVG on top of GL.

Well what other ways are there to render UI components ontop of an h264 stream then with IL and MMAL?

I have read the other thread. The two alternatives suggested are way out dated!

[6by9]

Well what other ways are there to render UI components ontop of an h264 stream then with IL and MMAL?

IL or MMAL, not IL and MMAL. They have the same functionality, so why use both?
I'm generally happy to support MMAL as it is a far simpler API to handle than IL with all the weird callbacks.

I have read the other thread. The two alternatives suggested are way out dated!

Learn GL then.
VG is obsolete and an abandoned spec, unchanged since 2011. (Conicidentally as is IL).
The existing VG stack you're using hasn't been modified since 2014 (possibly earlier), therefore feel free to call that outdated too.
If you wish to use an obsolete API, then don't dismiss things that haven't been updated recently.

[ktb92677]

Well what other ways are there to render UI components ontop of an h264 stream then with IL and MMAL?

IL or MMAL, not IL and MMAL. They have the same functionality, so why use both?
I'm generally happy to support MMAL as it is a far simpler API to handle than IL with all the weird callbacks.

I have read the other thread. The two alternatives suggested are way out dated!

Learn GL then.
VG is obsolete and an abandoned spec, unchanged since 2011. (Conicidentally as is IL).
The existing VG stack you're using hasn't been modified since 2014 (possibly earlier), therefore feel free to call that outdated too.
If you wish to use an obsolete API, then don't dismiss things that haven't been updated recently.

I was actually totally unaware that MMAL was able to decode h264 in a hardware accelerated fashion like IL is. A quick google search for such a sample program does not come up... is there any chance you could link me to one? I'd love to move away from IL.

Okay, I suppose it is time to learn GLES2 then!

EDIT: Also follow up question, if MMAL can do the exact same things as IL can and MMAL is well maintained, whereas IL has been abandoned why is the default example for hello_pi/hello_video still using IL?

EDIT2: Ah, found an example: https://github.com/raspberrypi/userland ... _basic_2.c. Could you confirm that this is a good example to go off of?

[jdonald]

Glad you narrowed down the problem. Even if OpenVG didn't have problematic interactions with other libraries, you wouldn't want to have our software locked out of the Pi 4. For a realtime streaming application where every millisecond counts, better to have it ready to go with more powerful hardware on arrival.

I want a hardware accelerated 2D renderer that is not dependent on X11 and also does not have me learning how to use shaders.

OpenGL has a fixed-function pipeline, so for straightforward 2D use-cases (like yours) you can use it without writing any shaders. OpenGL ES 1.0 also has a fixed-function pipeline: Interestingly it's deprecated on Raspbian Buster, but due to some odd quirk developers are still able to use it. I imagine it may be safe for a couple more years until Debian 11 Bullseye.

Neither of these APIs are dependent on X11. Sometimes it's a related framework (freeglut, glfw) that may require X11. For supported ways to render directly to the screen there are a few approaches, but the one I'm familiar with for the Pi 4 is to build SDL2 with --disable-video-x11 --disable-video-rpi --enable-video-kmsdrm then write your GL or GLES application on top of that. Also, SDL has its own 2D drawing API, SDL_Surface, that you might find easier.

[ktb92677]

Can I run ilclient and SDL2 at the same time? To have them overlay eachother almost?

[ktb92677]

I have horrible news. They system just crashed again. It seems to be able to survive a bit longer without the OpenVG interaction. But it has crashed in the same exact manner as before. Here's what it says in gdb

Code: Select all

(gdb) backtrace full
#0  0x751f30f0 in completion_thread () at /opt/vc/lib/libvchiq_arm.so
#1  0x76f68cb0 in vcos_thread_entry (arg=0x75206318 <vchiq_instance+16>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
        ret = <optimized out>
        thread = 0x75206318 <vchiq_instance+16>
#2  0x753cc494 in start_thread (arg=0x6fbfeb00) at pthread_create.c:486
        ret = <optimized out>
        pd = 0x6fbfeb00
        unwind_buf = 
              {cancel_jmp_buf = {{jmp_buf = {-1619936155, -2047697611, 1996482920, 1874848512, 1884525776, 338, 2130702306, 1874848512, 0, 1874847292, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#3  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Code: Select all

(gdb) thread apply all bt
                                                                          
Thread 154585 (Thread 0x70535b00 (LWP 32203)):                              
#0  0x7534f558 in clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:58           
#1  0x753cb180 in create_thread (pd=0x70535b68, attr=0x62efc414, stopped_start=0x62efc412, stackaddr=<optimized out>, thread_ran=0x0) at ../sysdeps/unix/sysv/linux/createthread.c:101
#2  0x00000000 in  ()                                                                 
                                                                                         
Thread 154533 (Thread 0x61efab00 (LWP 32128)):                                                 
#0  0x753176d0 in __GI___nanosleep (remaining=0x0, requested_time=0x61efa42c) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x753176d0 in __GI___nanosleep (requested_time=0x61efa42c, requested_time@entry=0x61efa424, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x75348fc0 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32                      
#3  0x00046ac0 in Check_Wifi_Battery(UTILITY_CORNER*) (utility_corner=0x1e71a8) at screens/corners/Utility_Corner.cpp:526
#4  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6                                             
#5  0x753cc494 in start_thread (arg=0x61efab00) at pthread_create.c:486                                               
#6  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73                                                    
                                                                                                                         
Thread 49 (Thread 0x606f7b00 (LWP 23913)):                                                                                  
#0  0x753d6088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x6481e3a4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753d6088 in do_futex_wait (sem=sem@entry=0x6481e3a4, abstime=0x0) at sem_waitcommon.c:115                                         
#2  0x753d61f4 in __new_sem_wait_slow (sem=0x6481e3a4, abstime=0x0) at sem_waitcommon.c:282                                                        
#3  0x76faa8bc in ilcs_execute_function_ex () at /opt/vc/lib/libopenmaxil.so                                                                         
#4  0x76fab8a4 in ilcs_pass_buffer () at /opt/vc/lib/libopenmaxil.so                                                                                               
#5  0x0002d370 in Render_H264_Frame(OPENMAX_H264_DECODER*, char*, int) (decoder=0x648139b8, buffer=0x213164 "", size=444) at openmax/OPENMAX_H264_DECODER.cpp:114                       
#6  0x000320d8 in Receive_Thread(LIVE_444_CLIENT*) (live=0x92be0) at live444/LIVE_444_CLIENT.cpp:28                                                                                       
#7  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6                                                                                                                      
#8  0x753cc494 in start_thread (arg=0x606f7b00) at pthread_create.c:486                                                                                                                   
#9  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 48 (Thread 0x60ef8b00 (LWP 23912)):
#0  0x753d6e98 in __lll_lock_wait (futex=futex@entry=0x752062ec <vchiq_lib_mutex>, private=0) at lowlevellock.c:46
#1  0x753cef44 in __GI___pthread_mutex_lock (mutex=0x752062ec <vchiq_lib_mutex>) at pthread_mutex_lock.c:80
#2  0x751f3d18 in vchiq_release_message () at /opt/vc/lib/libvchiq_arm.so
#3  0x76faaff4 in ilcs_task () at /opt/vc/lib/libopenmaxil.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x6481e1d8) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x753cc494 in start_thread (arg=0x60ef8b00) at pthread_create.c:486
#6  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 47 (Thread 0x616f9b00 (LWP 23911)):
#0  0x753d31a0 in futex_wait_cancelable (private=0, expected=0, futex_word=0x6481e1b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0x753d31a0 in __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x6481e188) at pthread_cond_wait.c:502
#2  0x753d31a0 in __pthread_cond_wait (cond=cond@entry=0x6481e188, mutex=0x0, mutex@entry=0x6481e16c) at pthread_cond_wait.c:655
#3  0x76f68d94 in _timer_thread (arg=0x6481e168) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:722
#4  0x753cc494 in start_thread (arg=0x616f9b00) at pthread_create.c:486
#5  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

--Type <RET> for more, q to quit, c to continue without paging--c
Thread 44 (Thread 0x62efcb00 (LWP 23908)):
#0  0x7534f558 in clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:58
#1  0x753cb180 in create_thread (pd=0x70535b68, attr=0x62efc414, stopped_start=0x62efc412, stackaddr=<optimized out>, thread_ran=0x70535b00) at ../sysdeps/unix/sysv/linux/createthread.c:101
#2  0x70535b70 in  ()

Thread 43 (Thread 0x636fdb00 (LWP 23907)):
#0  0x753cee48 in __GI___pthread_mutex_lock (mutex=0x6f6c8 <DEVICE_MANAGER::Keyboard_Events+40>) at pthread_mutex_lock.c:67
#1  0x76ed2ee0 in Send_Keyboard_Data(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#2  0x76ed3b28 in Send_Thread(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#3  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x753cc494 in start_thread (arg=0x636fdb00) at pthread_create.c:486
#5  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 38 (Thread 0x6d1fab00 (LWP 23812)):
#0  0x753486ac in __GI___select (timeout=0x6d1fa418, exceptfds=0x0, writefds=0x0, readfds=0x6d1fa458, nfds=12) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x753486ac in __GI___select (nfds=12, readfds=0x6d1fa458, writefds=0x0, exceptfds=0x0, timeout=0x6d1fa3e0) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x7580988c in Listen_Keyboard(KEYBOARD*) () at /usr/local/lib/libgui.so
#3  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x753cc494 in start_thread (arg=0x6d1fab00) at pthread_create.c:486
#5  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 16 (Thread 0x63efeb00 (LWP 23295)):
#0  0x75344cd0 in __GI___poll (timeout=1, nfds=1, fds=0x6a9abbd8) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x75344cd0 in __GI___poll (fds=0x6a9abbd8, nfds=1, timeout=1) at ../sysdeps/unix/sysv/linux/poll.c:26
#2  0x7580c16c in Listen_Mouse_Once(MOUSE*) () at /usr/local/lib/libgui.so
#3  0x76ed75b4 in Read_Mouse_Data(DEVICE_NODE*) () at /usr/local/lib/libUtil.so
#4  0x76ed9a60 in Read_Thread(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#5  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#6  0x753cc494 in start_thread (arg=0x63efeb00) at pthread_create.c:486

Thread 15 (Thread 0x675feb00 (LWP 23294)):
#0  0x753486ac in __GI___select (timeout=0x675f63f8, exceptfds=0x0, writefds=0x0, readfds=0x675f6458, nfds=9) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x753486ac in __GI___select (nfds=9, readfds=0x675f6458, writefds=0x0, exceptfds=0x0, timeout=0x675f63f0) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x76ed629c in Refresh_Thread_DEVICE_MANAGER(DEVICE_MANAGER*) () at /usr/local/lib/libUtil.so
#3  0x7553c9b0 in  () at /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x753cc494 in start_thread (arg=0x675feb00) at pthread_create.c:486

Thread 9 (Thread 0x6c9f9b00 (LWP 23249)):
#0  0x753d6088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x2349d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753d6088 in do_futex_wait (sem=sem@entry=0x2349d8, abstime=0x0) at sem_waitcommon.c:115
#2  0x753d61f4 in __new_sem_wait_slow (sem=0x2349d8, abstime=0x0) at sem_waitcommon.c:282
#3  0x758ce9c8 in mmal_component_action_thread_func () at /opt/vc/lib/libmmal_core.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x2348f0) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x753cc494 in start_thread (arg=0x6c9f9b00) at pthread_create.c:486
#6  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 6 (Thread 0x6e1fcb00 (LWP 23243)):
#0  0x753d6088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa665c <cecservice_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753d6088 in do_futex_wait (sem=sem@entry=0x76fa665c <cecservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753d61f4 in __new_sem_wait_slow (sem=0x76fa665c <cecservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f8ce98 in cecservice_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa6670 <cecservice_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x753cc494 in start_thread (arg=0x6e1fcb00) at pthread_create.c:486
#6  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 5 (Thread 0x6e9fdb00 (LWP 23242)):
#0  0x753d6088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa58cc <tvservice_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753d6088 in do_futex_wait (sem=sem@entry=0x76fa58cc <tvservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753d61f4 in __new_sem_wait_slow (sem=0x76fa58cc <tvservice_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f8bb28 in tvservice_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa58e0 <tvservice_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x753cc494 in start_thread (arg=0x6e9fdb00) at pthread_create.c:486
#6  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 4 (Thread 0x6f1feb00 (LWP 23241)):
#0  0x753d6088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76fa6758 <dispmanx_notify_available_event+24>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  0x753d6088 in do_futex_wait (sem=sem@entry=0x76fa6758 <dispmanx_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:115
#2  0x753d61f4 in __new_sem_wait_slow (sem=0x76fa6758 <dispmanx_notify_available_event+24>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76f903a4 in dispmanx_notify_func () at /opt/vc/lib/libbcm_host.so
#4  0x76f68cb0 in vcos_thread_entry (arg=0x76fa7498 <dispmanx_notify_task>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#5  0x753cc494 in start_thread (arg=0x6f1feb00) at pthread_create.c:486

Thread 3 (Thread 0x6fbfeb00 (LWP 23240)):
#0  0x751f30f0 in completion_thread () at /opt/vc/lib/libvchiq_arm.so
#1  0x76f68cb0 in vcos_thread_entry (arg=0x75206318 <vchiq_instance+16>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_pthreads.c:144
#2  0x753cc494 in start_thread (arg=0x6fbfeb00) at pthread_create.c:486
#3  0x7534f578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 1 (Thread 0x70539010 (LWP 23235)):
#0  0x753d7554 in __libc_recv (flags=256, len=17, buf=0x64801848, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/recv.c:26
#1  0x753d7554 in __libc_recv (fd=<optimized out>, buf=0x64801848, len=17, flags=256) at ../sysdeps/unix/sysv/linux/recv.c:23
#2  0x76ea89b0 in Client::Receive(char*, int) () at /usr/local/lib/libnetwork.so
#3  0x76eafe60 in Status_HERMES_CLIENT(HERMES_CLIENT*) () at /usr/local/lib/libnetwork.so
#4  0x0001ad4c in main() () at main.cpp:320

Please @6by9 I really need help on this one. I can't for the life of me solve this

[ktb92677]

Well what other ways are there to render UI components ontop of an h264 stream then with IL and MMAL?

IL or MMAL, not IL and MMAL. They have the same functionality, so why use both?
I'm generally happy to support MMAL as it is a far simpler API to handle than IL with all the weird callbacks.

I have read the other thread. The two alternatives suggested are way out dated!

Learn GL then.
VG is obsolete and an abandoned spec, unchanged since 2011. (Conicidentally as is IL).
The existing VG stack you're using hasn't been modified since 2014 (possibly earlier), therefore feel free to call that outdated too.
If you wish to use an obsolete API, then don't dismiss things that haven't been updated recently.

6by9 I need help really badly! Removing OpenVG did not make the error go away