abypaulvarghese
Posts: 8
Joined: Mon May 06, 2019 2:54 pm

v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 11:12 am

Hello All,

Context :
I have a PI4 with 2 GB RAM. (SWAP memory 100MB, GPU 512MB ) .I am developing an application that can capture H264 stream from IP cameras and put them into a ZMQ. I am using GStreamer and the pipeline looks like below :
pipelinestring = "rtspsrc location={0} name=m_rtspsrc ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! video/x-raw ! v4l2convert output-io-mode=5 capture-io-mode=4 ! video/x-raw, format=(string)BGR, width=(int)640, height=(int)480 ! videorate ! video/x-raw,framerate=1/1 ! appsink name=m_appsink sync=false".format(self.streamLink)

I am getting the frame from 4 different IP cameras, a separate process is started to fetch the frame from each cameras. ie I have 4 child process and a parent process. I have two other python programs to perform the analysis . As you might have noted, I plan to grab only one frame in a second from a camera. which means, I am expecting 4 frames in a second. Once I get the frame, I push it to a ZMQ to perform analysis on it.
Problem
The application works okay. Today I try to do an endurance testing of it. The application works okay for sometime (may be around 45 minutes?). The CPU utilization was okay, somewhere around 30-40 % (this is maximum usage, otherwise it used to be between 10-20). The temperature was also around 50-60 as per RPI. However I noticed the desktop flashes immediately and restarted (only the desktop GUI ) and prompted the Login ( this is not the system restart- system is configured not to prompt for password). I tried to launch my program and found all of them working wired, ie it was not connecting to cameras. I remember there was a message 'killed' if I remember correctly.

Analysis so far

I am a beginner in GStreamer all together. I started looked looking at the syslog. And I found symptoms of OOM. The following the portion from the syslog.
Incident 1 : Crash pointing out in v4l2h264dec .

Code: Select all

Jun 12 06:17:55 raspberrypi dhcpcd[436]: wlan0: deleting route to 192.168.1.0/24
Jun 12 06:17:55 raspberrypi dhcpcd[436]: wlan0: deleting default route via 192.168.1.1
Jun 12 06:18:00 raspberrypi systemd[1]: systemd-rfkill.service: Succeeded.
Jun 12 06:20:27 raspberrypi kernel: [  244.140910] usb 1-1.1: USB disconnect, device number 3
Jun 12 06:20:28 raspberrypi kernel: [  244.278554] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Jun 12 06:20:28 raspberrypi kernel: [  244.309785] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
Jun 12 06:20:28 raspberrypi kernel: [  244.309795] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 12 06:20:28 raspberrypi kernel: [  244.309803] usb 2-1: Product: VSC Loopback Device
Jun 12 06:20:28 raspberrypi kernel: [  244.309810] usb 2-1: Manufacturer: Intel Corporation
Jun 12 06:20:28 raspberrypi kernel: [  244.309817] usb 2-1: SerialNumber: DE2856CFB41C00
Jun 12 06:20:28 raspberrypi mtp-probe: checking bus 2, device 2: "/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1"
Jun 12 06:20:28 raspberrypi mtp-probe: bus: 2, device: 2 was not an MTP device
Jun 12 06:20:28 raspberrypi mtp-probe: checking bus 2, device 2: "/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1"
Jun 12 06:20:28 raspberrypi mtp-probe: bus: 2, device: 2 was not an MTP device
Jun 12 06:20:31 raspberrypi kernel: [  247.516048] v3d fec00000.v3d: MMU error from client L2T (0) at 0x38a1000, pte invalid
Jun 12 06:25:01 raspberrypi CRON[2465]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jun 12 06:31:33 raspberrypi systemd[1]: Starting Cleanup of Temporary Directories...
Jun 12 06:31:33 raspberrypi systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jun 12 06:31:33 raspberrypi systemd[1]: Started Cleanup of Temporary Directories.
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205093] v4l2h264dec0:sr invoked oom-killer: gfp_mask=0x400dc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_ZERO), order=0, oom_score_adj=0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205106] CPU: 2 PID: 1857 Comm: v4l2h264dec0:sr Tainted: G         C        5.10.17-v7l+ #1414
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205111] Hardware name: BCM2711
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205115] Backtrace: 
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205137] [<c0b63050>] (dump_backtrace) from [<c0b633e4>] (show_stack+0x20/0x24)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205144]  r7:ffffffff r6:00000000 r5:60000113 r4:c12e69fc
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205152] [<c0b633c4>] (show_stack) from [<c0b67774>] (dump_stack+0xcc/0xf8)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205159] [<c0b676a8>] (dump_stack) from [<c0b65a04>] (dump_header+0x64/0x208)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205165]  r10:c120518c r9:00400dc2 r8:00000000 r7:c0e3709c r6:c6605800 r5:c55dbe00
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205170]  r4:c46d1de8 r3:8f05e476
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205180] [<c0b659a0>] (dump_header) from [<c03b6cd0>] (oom_kill_process+0x1b4/0x1c0)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205185]  r7:c0e3709c r6:c46d1de8 r5:c55dc380 r4:c55dbe00
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205192] [<c03b6b1c>] (oom_kill_process) from [<c03b7848>] (out_of_memory+0x2b8/0x390)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205197]  r7:c1208580 r6:c1205048 r5:c55dbe00 r4:c46d1de8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205205] [<c03b7590>] (out_of_memory) from [<c040a8bc>] (__alloc_pages_nodemask+0x7ec/0x1184)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205210]  r7:c1336790 r6:00001400 r5:0000174f r4:00000000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205220] [<c040a0d0>] (__alloc_pages_nodemask) from [<c03ec934>] (__pte_alloc+0x34/0x168)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205226]  r10:c1205048 r9:00000040 r8:00000255 r7:00000000 r6:c70b98d8 r5:c3a4b000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205230]  r4:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205238] [<c03ec900>] (__pte_alloc) from [<c03f0068>] (handle_mm_fault+0x100c/0x1124)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205243]  r7:00000000 r6:070b9003 r5:00000000 r4:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205251] [<c03ef05c>] (handle_mm_fault) from [<c0b739d4>] (do_page_fault+0x144/0x314)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205256]  r10:00000255 r9:00000800 r8:c3a4b048 r7:c3a4b000 r6:00000a06 r5:a371e000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205260]  r4:c46d1fb0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205266] [<c0b73890>] (do_page_fault) from [<c0b73c84>] (do_translation_fault+0xe0/0x10c)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205272]  r10:00000000 r9:b6dcb258 r8:00000a06 r7:a371e000 r6:c0b73ba4 r5:00000a06
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205276]  r4:c120aef8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205283] [<c0b73ba4>] (do_translation_fault) from [<c0214c6c>] (do_DataAbort+0x40/0xc0)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205289]  r9:b6dcb258 r8:c46d1fb0 r7:a371e000 r6:c0b73ba4 r5:00000a06 r4:c120aef8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205296] [<c0214c2c>] (do_DataAbort) from [<c0200da4>] (__dabt_usr+0x44/0x60)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205300] Exception stack(0xc46d1fb0 to 0xc46d1ff8)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205305] 1fa0:                                     a371e000 000e2002 00000000 a371e008
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205311] 1fc0: a9200010 000e2000 0001c201 000e1018 b6dcb000 b6dcb258 00000000 fffff000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205316] 1fe0: b6dcb26c a49747b0 b6dcb258 b6cf18d8 40000010 ffffffff
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205322]  r8:30c5387d r7:30c5383d r6:ffffffff r5:40000010 r4:b6cf18d8
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205326] Mem-Info:
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338] active_anon:42279 inactive_anon:244363 isolated_anon:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338]  active_file:619 inactive_file:5129 isolated_file:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338]  unevictable:6795 dirty:0 writeback:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338]  slab_reclaimable:3594 slab_unreclaimable:11751
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338]  mapped:6721 shmem:8017 pagetables:2109 bounce:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205338]  free:5252 free_pcp:0 free_cma:2
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205349] Node 0 active_anon:169116kB inactive_anon:977452kB active_file:2476kB inactive_file:20516kB unevictable:27180kB isolated(anon):0kB isolated(file):0kB mapped:26884kB dirty:0kB writeback:0kB shmem:32068kB writeback_tmp:0kB kernel_stack:2544kB all_unreclaimable? no
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205360] DMA free:18136kB min:20480kB low:24576kB high:28672kB reserved_highatomic:0KB active_anon:0kB inactive_anon:158976kB active_file:932kB inactive_file:4600kB unevictable:0kB writepending:0kB present:524288kB managed:421900kB mlocked:0kB pagetables:848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:8kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205366] lowmem_reserve[]: 0 0 1024 0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205402] HighMem free:2872kB min:31820kB low:42000kB high:52180kB reserved_highatomic:0KB active_anon:169116kB inactive_anon:818476kB active_file:1448kB inactive_file:15944kB unevictable:27180kB writepending:0kB present:1048576kB managed:1048576kB mlocked:16kB pagetables:7588kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205408] lowmem_reserve[]: 0 0 0 0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205439] DMA: 604*4kB (UMEC) 272*8kB (UMEC) 206*16kB (UME) 103*32kB (UME) 82*64kB (UME) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18224kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205543] HighMem: 180*4kB (UM) 67*8kB (UM) 37*16kB (UM) 30*32kB (UM) 7*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3256kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205640] 13826 total pagecache pages
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205647] 23 pages in swap cache
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205654] Swap cache stats: add 25651, delete 25628, find 1089/1106
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205660] Free swap  = 0kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205666] Total swap = 102396kB
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205672] 393216 pages RAM
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205678] 262144 pages HighMem/MovableOnly
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205684] 25597 pages reserved
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205690] 65536 pages cma reserved
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205696] Tasks state (memory values in pages):
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205702] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205718] [    113]     0   113     5352       19   167936      224             0 systemd-journal
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205726] [    148]     0   148     4651       49    49152      195         -1000 systemd-udevd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205736] [    322]   100   322     5604       17    61440      146             0 systemd-timesyn
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205744] [    366]   108   366     1442       34    40960       43             0 avahi-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205752] [    370]     0   370     1987       15    45056       30             0 cron
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205759] [    371]     0   371    16305       71    90112      397             0 udisksd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205767] [    382]   108   382     1442       14    32768       49             0 avahi-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205775] [    383]     0   383     6914        8    40960       15             0 rngd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205783] [    387]     0   387     2942       14    49152      137             0 alsactl
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205791] [    405]   104   405     1710       71    40960       96          -900 dbus-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205798] [    413]     0   413     2686       13    45056       88             0 wpa_supplicant
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205806] [    414]     0   414     6378        0    57344      219             0 rsyslogd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205814] [    419]     0   419     3261       18    53248      151             0 systemd-logind
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205821] [    426] 65534   426     1080        6    36864       38             0 thd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205829] [    436]     0   436      726        8    28672       93             0 dhcpcd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205836] [    487]     0   487     2157        0    45056       56             0 vncserver-x11-s
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205844] [    496]     0   496     9827       78    90112     1344             0 vncserver-x11-c
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205851] [    497]     0   497    10358        1    77824      427             0 polkitd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205859] [    503]     0   503    10025        1    65536      198             0 lightdm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205867] [    513]     0   513     2682        0    49152      143         -1000 sshd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205874] [    524]     0   524     2751       23    45056      121             0 wpa_supplicant
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205882] [    542]     0   542    42377     3436   290816     3468             0 Xorg
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205890] [    543]     0   543     1405        2    40960      111             0 login
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205898] [    553]     0   553     2870        1    45056      123             0 vncagent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205905] [    561]     0   561     8125        0    69632      214             0 lightdm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205913] [    566]  1000   566     3682        2    53248      255             0 systemd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205920] [    567]  1000   567     4185       14    57344      401             0 (sd-pam)
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205928] [    577]  1000   577    14052        2    98304      317             0 lxsession
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205936] [    585]  1000   585     1637        0    40960      110             0 dbus-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205943] [    609]  1000   609     1130       10    32768       62             0 ssh-agent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205951] [    620]  1000   620    10852        1    69632      166             0 gvfsd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205958] [    625]  1000   625    14189        0    77824      188             0 gvfsd-fuse
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205966] [    632]  1000   632    15583        2    94208      493             0 openbox
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205974] [    634]  1000   634    11782        1    94208      275             0 lxpolkit
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205981] [    643]  1000   643   105838     1025   217088     1035             0 lxpanel
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205989] [    644]  1000   644    26595     1020   143360      975             0 pcmanfm
Jun 12 07:09:03 raspberrypi kernel: [ 3159.205996] [    652]  1000   652     1130        1    32768       71             0 ssh-agent
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206004] [    654]  1000   654     1218        0    36864       67             0 xcompmgr
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206012] [    668]  1000   668    12973        0   126976     3302             0 applet.py
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206019] [    671]  1000   671    91936        0    86016      316             0 pulseaudio
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206027] [    686]   110   686     5797        1    49152       50             0 rtkit-daemon
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206035] [    693]  1000   693     2189        1    45056      303             0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206042] [    706]  1000   706    19865        0    94208      344             0 gvfs-udisks2-vo
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206050] [    712]  1000   712     8006        1    86016      541             0 vncserverui
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206058] [    732]  1000   732    14007        1    94208      202             0 gvfs-afc-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206065] [    734]  1000   734     7174        0    57344      150             0 menu-cached
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206072] [    742]  1000   742    10032        0    69632      133             0 gvfs-mtp-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206080] [    746]  1000   746     7934       30    90112      378             0 vncserverui
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206088] [    747]  1000   747    10424        0    65536      161             0 gvfs-gphoto2-vo
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206095] [    751]  1000   751    10026        0    61440      122             0 gvfs-goa-volume
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206103] [    761]     0   761      535        1    28672       31             0 hciattach
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206111] [    767]     0   767     2384        0    45056       94             0 bluetoothd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206118] [    771]  1000   771    14239        0    90112      700             0 gvfsd-trash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206126] [   1046]     0  1046     6910        0    69632      231             0 cupsd
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206134] [   1048]     0  1048    10124        1    77824      272             0 cups-browsed
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206142] [   1295]  1000  1295    42312     1756   290816     2000             0 lxterminal
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206149] [   1304]  1000  1304     2226        2    40960      336             0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206157] [   1328]  1000  1328     2226       55    36864      278             0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206173] [   1347]  1000  1347     2226       59    40960      279             0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206210] [   1376]  1000  1376     2193       14    40960      284             0 bash
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206218] [   1544]  1000  1544    47592     3750   233472      282             0 python
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206225] [   1670]  1000  1670    77117    10044   344064        0             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206233] [   1802]  1000  1802    44780     4787   225280        0             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206241] [   1810]  1000  1810    89999    13550   385024        7             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206248] [   1839]  1000  1839    84168     7402   331776        5             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206256] [   1859]  1000  1859   325845   249318  2273280        5             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206263] [   1887]  1000  1887    83785     8281   331776        5             0 python3
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206271] [   1926]  1000  1926     2022      132    40960        0             0 htop
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206282] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python3,pid=1859,uid=1000
Jun 12 07:09:03 raspberrypi kernel: [ 3159.206371] Out of memory: Killed process 1859 (python3) total-vm:1303380kB, anon-rss:997260kB, file-rss:0kB, shmem-rss:12kB, UID:1000 pgtables:2220kB oom_score_adj:0
Jun 12 07:09:03 raspberrypi kernel: [ 3159.403435] oom_reaper: reaped process 1859 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:8kB
Jun 12 07:16:31 raspberrypi rngd[383]: stats: bits received from HRNG source: 80064
Jun 12 07:16:31 raspberrypi rngd[383]: stats: bits sent to kernel pool: 33824
Jun 12 07:16:31 raspberrypi rngd[383]: stats: entropy added to kernel pool: 33824
My observations :
The Free Swap memory is noted as 0 KB.

Questions : What is causing the crash ? is it because the Swap memory becoming 0 causing the crash? Could someone help me to understand how should I handle it ? is it something that I need to take care in the pipeline or in the python program?

Kindly note: Happy to share the complete source code on Github if required.

Thank You
Aby

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11487
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 1:50 pm

v4l2h264dec wanted some memory and there was none left. That triggers the Out Of Memory killer to try and free up some memory.
It's not the fault of v4l2h264dec, but probably indicative that something in the system has a memory leak.

gpu_mem of 512MB is generally excessive, although it does depend on what level your H264 streams are. 256MB should be sufficient.
You may wish to check how much CMA heap is available as V4L2 image buffers are allocated from there, as are all desktop graphics resources. "dmesg | grep -i cma" is probably the quickest check if done soon after boot.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

abypaulvarghese
Posts: 8
Joined: Mon May 06, 2019 2:54 pm

Re: v4l2h264dec invoked oom-killer : Hardware name: BCM2711 - application crashed

Sat Jun 12, 2021 4:35 pm

Thank you very much 6by9.

The overall process is given below:
App 1: Grab a frame from each of the cameras (4 cameras in total) in every second ->Push it to ZMQ (PUB - publisher mode) . Total 4 Publishers.
App 2: Grab the frame from ZMQ (SUB) by connecting to 4 publishers. Perform an AI inference using Intel NCS which is connected to one of the USB ports. The result is pushed into another ZMQ.
App3 : Grabs the data from ZMQ of App2 . Log / communicate the result over web.

I ran the command and the output is below.

Code: Select all

pi@raspberrypi:~ $ dmesg | grep -i cma
[    0.000000] Reserved memory: created CMA memory pool at 0x000000000fc00000, size 256 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Memory: 1206284K/1572864K available (10240K kernel code, 1358K rwdata, 3160K rodata, 2048K init, 890K bss, 104436K reserved, 262144K cma-reserved, 1048576K highmem)
[    4.917746] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[    4.923118] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
I observed similar issue once again, however this time it was clearly from one of the python application as per the syslog. The call stack was pointing to tcp_sendmsg() as per syslog.
Hence I my hypothesis is:
Hypothesis 1: The App2 (which is supposed to process the frames from the Queue ) is crashed due to some reason(may be an exception from NCS or other reasons). The 4 PUBs continued to push data into the Queue and since the SUB was down the memory kept on growing. It failed at one stage where there was no more memory left to add new item to the Queue.
I am planning to test the above hypothesis and planning to change the GPU memory to 256 MB. thoughts?


Thank you
Aby

Return to “OpenMAX”