Moving Linux kernel to 4.19 [Already moved for many months now (in january 2020)]

[tonyhain]

Didn't see the comment about stale hostapd as I was putting my previous comment together. After seeing that though I did a quick check and found the one I was using is even older

Code: Select all

# hostapd -v 
hostapd v2.3
User space daemon for IEEE 802.11 AP management,
IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
Copyright (c) 2002-2014, Jouni Malinen <j@w1.fi> and contributors

will contact cyoung about updating it. Since the stratux distribution is still 4.9, it is unlikely hostapd will get updated on it until AP mode is stable for more than a couple of weeks on either 4.14 or 4.19.

In the FWIW department, after 1 hr 5 min uptime, the AP mode dropped in 4.14.98 just like it had been in 4.9

Code: Select all

[ 3935.209690] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
[ 4007.328004] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4009.887982] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4009.887999] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[ 4140.768549] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4140.768564] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 4143.328516] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4143.328532] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
[ 4212.368820] brcmfmac: brcmf_netdev_wait_pend8021x: Timed out waiting for no pending 802.1x packets

modprobe bounce had no effect

Code: Select all

[ 4484.799515] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.799522] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.799645] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.810414] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.810430] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 4484.991871] usbcore: deregistering interface driver brcmfmac
[ 4488.815263] brcmfmac: F1 signature read @0x18000000=0x1541a9a6
[ 4488.823261] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43430-sdio.bin for chip 0x00a9a6(43430) rev 0x000001
[ 4488.823570] usbcore: registered new interface driver brcmfmac
[ 4491.489439] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 4491.492721] brcmfmac: brcmf_c_get_clm_name: retrieving revision info failed (-110)
[ 4491.492730] brcmfmac: brcmf_c_process_clm_blob: get CLM blob file name failed (-110)
[ 4491.492738] brcmfmac: brcmf_c_preinit_dcmds: download CLM blob file failed, -110
[ 4491.492745] brcmfmac: brcmf_bus_started: failed: -110
[ 4491.492767] brcmfmac: brcmf_sdio_firmware_callback: dongle is not responding

Code: Select all

# iw phy
# echo -n 3f300000.mmc > /sys/devices/platform/soc/3f300000.mmc/driver/unbind
# echo -n 3f300000.mmc > /sys/bus/platform/drivers/mmc-bcm2835/bind
# iw phy
# dmesg
...
[ 5039.299639] ------------[ cut here ]------------
[ 5039.299680] WARNING: CPU: 3 PID: 984 at kernel/irq/manage.c:1542 __free_irq+0xd0/0x31c
[ 5039.299689] Trying to free already-free IRQ 92
[ 5039.299696] Modules linked in: brcmfmac brcmutil cfg80211 rfkill ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables cdc_acm snd_bcm2835(C) snd_pcm snd_timer snd i2c_bcm2835 joydev rpi_backlight uio_pdrv_genirq rpi_ft5406 uio fixed evdev i2c_dev i2c_bcm2708 ipv6 [last unloaded: brcmutil]
[ 5039.299933] CPU: 3 PID: 984 Comm: bash Tainted: G        WC      4.14.98-v7+ #1200
[ 5039.299943] Hardware name: BCM2835
[ 5039.299984] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
[ 5039.300017] [<8010c174>] (show_stack) from [<8078cf04>] (dump_stack+0xd4/0x118)
[ 5039.300039] [<8078cf04>] (dump_stack) from [<8011dd70>] (__warn+0xf8/0x110)
[ 5039.300057] [<8011dd70>] (__warn) from [<8011ddd0>] (warn_slowpath_fmt+0x48/0x50)
[ 5039.300079] [<8011ddd0>] (warn_slowpath_fmt) from [<801773ac>] (__free_irq+0xd0/0x31c)
[ 5039.300114] [<801773ac>] (__free_irq) from [<8017769c>] (free_irq+0x4c/0x94)
[ 5039.300148] [<8017769c>] (free_irq) from [<8017b554>] (devm_irq_release+0x1c/0x20)
[ 5039.300177] [<8017b554>] (devm_irq_release) from [<8054b5b0>] (release_nodes+0x17c/0x1e4)
[ 5039.300215] [<8054b5b0>] (release_nodes) from [<8054c054>] (devres_release_all+0x40/0x5c)
[ 5039.300241] [<8054c054>] (devres_release_all) from [<8054860c>] (device_release_driver_internal+0x164/0x1ec)
[ 5039.300259] [<8054860c>] (device_release_driver_internal) from [<805486b4>] (device_release_driver+0x20/0x24)
[ 5039.300273] [<805486b4>] (device_release_driver) from [<80546a00>] (unbind_store+0x88/0x108)
[ 5039.300287] [<80546a00>] (unbind_store) from [<80545f60>] (drv_attr_store+0x30/0x3c)
[ 5039.300304] [<80545f60>] (drv_attr_store) from [<8030acc0>] (sysfs_kf_write+0x54/0x58)
[ 5039.300337] [<8030acc0>] (sysfs_kf_write) from [<8030a334>] (kernfs_fop_write+0xdc/0x1c0)
[ 5039.300360] [<8030a334>] (kernfs_fop_write) from [<8028bee4>] (__vfs_write+0x38/0x138)
[ 5039.300378] [<8028bee4>] (__vfs_write) from [<8028c1ac>] (vfs_write+0xb4/0x1bc)
[ 5039.300410] [<8028c1ac>] (vfs_write) from [<8028c3fc>] (SyS_write+0x54/0xb0)
[ 5039.300434] [<8028c3fc>] (SyS_write) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5039.300445] ---[ end trace 338168505c054132 ]---
[ 5041.554415] mmc-bcm2835 3f300000.mmc: mmc_debug:0 mmc_debug2:0
[ 5041.554428] mmc-bcm2835 3f300000.mmc: DMA channel allocated
[ 5041.623849] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.626028] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.626672] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.626689] mmc1: error -22 whilst initialising SDIO card
[ 5041.704652] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.707501] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.708316] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.708332] mmc1: error -22 whilst initialising SDIO card
[ 5041.798438] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.802761] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.803953] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.803970] mmc1: error -22 whilst initialising SDIO card
[ 5041.899864] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.907898] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.910194] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.910211] mmc1: error -22 whilst initialising SDIO card

This would appear to have nothing to do with hostapd as the device simply disappears out from under it and can't be reestablished. Will swap to the 7.45.41.46 sdio.bin and try again.

[dom]

Because of the many complains about broken AP mode, i want to link to my post from November:
https://www.raspberrypi.org/forums/view ... 5#p1396409

Since this won't be fixed in mainline, we have two options:
1) update hostapd to a more recent version (e.g. from Buster)
2) revert the mentioned commit

Thanks for reminder. I'll check with colleagues which option is preferred.
(Note: we're planning on switching raspbian to buster as soon as it's stable,
so hopefully this issue will vanish, but that is likely a few months off, so a temporary solution is wanted).

[dom]

Since this won't be fixed in mainline, we have two options:
1) update hostapd to a more recent version (e.g. from Buster)
2) revert the mentioned commit

hostapd in raspbian has been updated to version from buster. Hopefully this resolved the issue.

[lowflyer]

@dom Thanks!!! You are a star!

hostapd in raspbian has been updated to version from buster. Hopefully this resolved the issue.

Starting with my working AP using 4.14.98-v7+ #1200, I did this...

Code: Select all

sudo apt update
sudo apt upgrade
sudo systemctl enable hostapd.service
sudo shutdown -r now

And the new version of hostapd (v2.6) works with the 4.14.98-v7+ #1200 kernel.

Then:

Code: Select all

sudo rpi-update
sudo shutdown -r now

And the AP works with kernel 4.19.25-v7+ #1205.

Well done!!

[kozman]

Good to hear the updated Hostpad version is helping.

[dom]

And the AP works with kernel 4.19.25-v7+ #1205.

Good to hear!

[lems]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Code: Select all

[ 1192.086591] e4defrag invoked oom-killer: gfp_mask=0x6000d0(GFP_KERNEL|__GFP_RECLAIMABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1192.086604] e4defrag cpuset=/ mems_allowed=0
[ 1192.086630] CPU: 2 PID: 2149 Comm: e4defrag Tainted: G         C O      4.19.23-v7+ #1203
[ 1192.086634] Hardware name: BCM2835
[ 1192.086668] [<80111ea8>] (unwind_backtrace) from [<8010d440>] (show_stack+0x20/0x24)
[ 1192.086685] [<8010d440>] (show_stack) from [<8080c880>] (dump_stack+0xd4/0x118)
[ 1192.086703] [<8080c880>] (dump_stack) from [<8023a7f0>] (dump_header+0x80/0x250)
[ 1192.086718] [<8023a7f0>] (dump_header) from [<80239b68>] (oom_kill_process+0x358/0x3a8)
[ 1192.086731] [<80239b68>] (oom_kill_process) from [<8023a498>] (out_of_memory+0x134/0x36c)
[ 1192.086746] [<8023a498>] (out_of_memory) from [<802408c0>] (__alloc_pages_nodemask+0x1024/0x1178)
[ 1192.086762] [<802408c0>] (__alloc_pages_nodemask) from [<802901d0>] (new_slab+0x520/0x6b4)
[ 1192.086775] [<802901d0>] (new_slab) from [<80292190>] (___slab_alloc.constprop.11+0x454/0x594)
[ 1192.086788] [<80292190>] (___slab_alloc.constprop.11) from [<80292314>] (__slab_alloc.constprop.10+0x44/0x90)
[ 1192.086801] [<80292314>] (__slab_alloc.constprop.10) from [<80292ae0>] (kmem_cache_alloc+0x20c/0x23c)
[ 1192.086815] [<80292ae0>] (kmem_cache_alloc) from [<802c3450>] (__d_alloc+0x34/0x1f0)
[ 1192.086830] [<802c3450>] (__d_alloc) from [<802c362c>] (d_alloc+0x20/0x88)
[ 1192.086843] [<802c362c>] (d_alloc) from [<802c3ab0>] (d_alloc_parallel+0x68/0x500)
[ 1192.086858] [<802c3ab0>] (d_alloc_parallel) from [<802b3fdc>] (__lookup_slow+0x6c/0x174)
[ 1192.086873] [<802b3fdc>] (__lookup_slow) from [<802b4124>] (lookup_slow+0x40/0x54)
[ 1192.086885] [<802b4124>] (lookup_slow) from [<802b6f08>] (walk_component+0x1f8/0x2f8)
[ 1192.086896] [<802b6f08>] (walk_component) from [<802b75b4>] (path_lookupat+0x78/0x214)
[ 1192.086908] [<802b75b4>] (path_lookupat) from [<802b90d4>] (filename_lookup+0xac/0x11c)
[ 1192.086919] [<802b90d4>] (filename_lookup) from [<802b9254>] (user_path_at_empty+0x50/0x58)
[ 1192.086931] [<802b9254>] (user_path_at_empty) from [<802ad820>] (vfs_statx+0x7c/0xe8)
[ 1192.086944] [<802ad820>] (vfs_statx) from [<802ae444>] (sys_fstatat64+0x40/0x70)
[ 1192.086958] [<802ae444>] (sys_fstatat64) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[ 1192.086963] Exception stack(0xa604bfa8 to 0xa604bff0)
[ 1192.086974] bfa0:                   7e9b3388 7e9b1024 00000007 01cd7f8b 7e9b0f78 00000100
[ 1192.086984] bfc0: 7e9b3388 7e9b1024 7e9b0f78 00000147 7e9b3388 7e9b3388 7e9b33c0 7e9b344c
[ 1192.086991] bfe0: 0000001c 7e9b0f58 76e78288 76e74cc0
[ 1192.086997] Mem-Info:
[ 1192.087017] active_anon:13885 inactive_anon:2626 isolated_anon:0
[ 1192.087017]  active_file:369 inactive_file:448 isolated_file:0
[ 1192.087017]  unevictable:438 dirty:0 writeback:0 unstable:0
[ 1192.087017]  slab_reclaimable:211896 slab_unreclaimable:9973
[ 1192.087017]  mapped:1277 shmem:2670 pagetables:608 bounce:0

It has been working perfectly for years including kernel 4.14.98. External disk has fair amount of files - 2GB in total, 2.5 million files.

I experienced this, too. I'm running a RPi 1 B headless, with gpu_mem_512=16. Upgrading to 4.19 resulted in rsnapshot no longer being able to complete (I have two USB disks attached via an active hub, both formatted using ext4). The OOM killer would start killing all kinds of processes, including rsync. Rolling back to 4.14. This setup has been working flawlessly the last years (first on Void Linux, now on Devuan).

edit: I also use a 1GB swap file that is located on one of the disks.

[Ernst]

I have decided to give hostapd another chance and it looks very promising.
See https://www.raspberrypi.org/forums/view ... 6#p1438636

[muhviehstarr]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Code: Select all

[ 1192.086591] e4defrag invoked oom-killer: gfp_mask=0x6000d0(GFP_KERNEL|__GFP_RECLAIMABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1192.086604] e4defrag cpuset=/ mems_allowed=0
[ 1192.086630] CPU: 2 PID: 2149 Comm: e4defrag Tainted: G         C O      4.19.23-v7+ #1203
[ 1192.086634] Hardware name: BCM2835
[ 1192.086668] [<80111ea8>] (unwind_backtrace) from [<8010d440>] (show_stack+0x20/0x24)
[ 1192.086685] [<8010d440>] (show_stack) from [<8080c880>] (dump_stack+0xd4/0x118)
[ 1192.086703] [<8080c880>] (dump_stack) from [<8023a7f0>] (dump_header+0x80/0x250)
[ 1192.086718] [<8023a7f0>] (dump_header) from [<80239b68>] (oom_kill_process+0x358/0x3a8)
[ 1192.086731] [<80239b68>] (oom_kill_process) from [<8023a498>] (out_of_memory+0x134/0x36c)
[ 1192.086746] [<8023a498>] (out_of_memory) from [<802408c0>] (__alloc_pages_nodemask+0x1024/0x1178)
[ 1192.086762] [<802408c0>] (__alloc_pages_nodemask) from [<802901d0>] (new_slab+0x520/0x6b4)
[ 1192.086775] [<802901d0>] (new_slab) from [<80292190>] (___slab_alloc.constprop.11+0x454/0x594)
[ 1192.086788] [<80292190>] (___slab_alloc.constprop.11) from [<80292314>] (__slab_alloc.constprop.10+0x44/0x90)
[ 1192.086801] [<80292314>] (__slab_alloc.constprop.10) from [<80292ae0>] (kmem_cache_alloc+0x20c/0x23c)
[ 1192.086815] [<80292ae0>] (kmem_cache_alloc) from [<802c3450>] (__d_alloc+0x34/0x1f0)
[ 1192.086830] [<802c3450>] (__d_alloc) from [<802c362c>] (d_alloc+0x20/0x88)
[ 1192.086843] [<802c362c>] (d_alloc) from [<802c3ab0>] (d_alloc_parallel+0x68/0x500)
[ 1192.086858] [<802c3ab0>] (d_alloc_parallel) from [<802b3fdc>] (__lookup_slow+0x6c/0x174)
[ 1192.086873] [<802b3fdc>] (__lookup_slow) from [<802b4124>] (lookup_slow+0x40/0x54)
[ 1192.086885] [<802b4124>] (lookup_slow) from [<802b6f08>] (walk_component+0x1f8/0x2f8)
[ 1192.086896] [<802b6f08>] (walk_component) from [<802b75b4>] (path_lookupat+0x78/0x214)
[ 1192.086908] [<802b75b4>] (path_lookupat) from [<802b90d4>] (filename_lookup+0xac/0x11c)
[ 1192.086919] [<802b90d4>] (filename_lookup) from [<802b9254>] (user_path_at_empty+0x50/0x58)
[ 1192.086931] [<802b9254>] (user_path_at_empty) from [<802ad820>] (vfs_statx+0x7c/0xe8)
[ 1192.086944] [<802ad820>] (vfs_statx) from [<802ae444>] (sys_fstatat64+0x40/0x70)
[ 1192.086958] [<802ae444>] (sys_fstatat64) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[ 1192.086963] Exception stack(0xa604bfa8 to 0xa604bff0)
[ 1192.086974] bfa0:                   7e9b3388 7e9b1024 00000007 01cd7f8b 7e9b0f78 00000100
[ 1192.086984] bfc0: 7e9b3388 7e9b1024 7e9b0f78 00000147 7e9b3388 7e9b3388 7e9b33c0 7e9b344c
[ 1192.086991] bfe0: 0000001c 7e9b0f58 76e78288 76e74cc0
[ 1192.086997] Mem-Info:
[ 1192.087017] active_anon:13885 inactive_anon:2626 isolated_anon:0
[ 1192.087017]  active_file:369 inactive_file:448 isolated_file:0
[ 1192.087017]  unevictable:438 dirty:0 writeback:0 unstable:0
[ 1192.087017]  slab_reclaimable:211896 slab_unreclaimable:9973
[ 1192.087017]  mapped:1277 shmem:2670 pagetables:608 bounce:0

It has been working perfectly for years including kernel 4.14.98. External disk has fair amount of files - 2GB in total, 2.5 million files.

I experienced this, too. I'm running a RPi 1 B headless, with gpu_mem_512=16. Upgrading to 4.19 resulted in rsnapshot no longer being able to complete (I have two USB disks attached via an active hub, both formatted using ext4). The OOM killer would start killing all kinds of processes, including rsync. Rolling back to 4.14. This setup has been working flawlessly the last years (first on Void Linux, now on Devuan).

edit: I also use a 1GB swap file that is located on one of the disks.

Tested this on rpi3b+ with (custom) 4.19 kernel under archlinux

after a minute the pi freezed, had to reboot via power cable

[dom]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

[dariuszb]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

Thank you. It makes a lot of sense now. Since 4.19 I have multiple out of memory situations when dealing with ext4 eternal drive operations (it all worked in 4.14).Not only e4defrag but also simple operations like du or tar as soon as applied to folders with many entries - I realize that there are always some limits and 1GB of RAM is going to hit them rather early but this was a bit too early:)

[kozman]

Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.

[jamesh]

Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.

Why would you expect noticeably more performance? The Linux kernel is pretty well optimised already, so releases are unlikely to provide much in the way of improved oomph. In fact, given the increased emphasis on security, that might require decreases in performance due to extra code requried in critical paths.

[kozman]

Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.

Why would you expect noticeably more performance? The Linux kernel is pretty well optimised already, so releases are unlikely to provide much in the way of improved oomph. In fact, given the increased emphasis on security, that might require decreases in performance due to extra code requried in critical paths.

Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance. Only some Spectre stuff affected Arm and in only the most minor of ways. Most of the mitigation perf drops came from non-Arch specific fixes in the kernel. So while I did expect some perf loss going from 4.14 to 4.19, some perf drops were quite dramatic in what the Phoronix benchmarks showed. Which is why it should be interesting to see what Buster will do against Stretch once it goes gold.

[6by9]

Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance.

Iff you run the mainline kernel then this would be important. The majority of people don't, therefore it makes almost no difference at all.

Mainline is missing a fair number of the features that are present in the Pi downstream kernel, such as the FIQ handling for the USB controller.

[kozman]

Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance.

Iff you run the mainline kernel then this would be important. The majority of people don't, therefore it makes almost no difference at all.

Mainline is missing a fair number of the features that are present in the Pi downstream kernel, such as the FIQ handling for the USB controller.

Gotcha.

[dariuszb]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

[kozman]

With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.

[jamesh]

Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.

We are currently discussing what to do with this internally. Upstream are pretty slow at fixing this, although there are possible patches that we may take and test.

[kozman]

Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.

We are currently discussing what to do with this internally. Upstream are pretty slow at fixing this, although there are possible patches that we may take and test.

"Pretty slow" can sometimes be glacial with upstream adoption. I *totally* know what you mean. Months and years have passed by in the past. But seeing as how this affects quite a few moving parts, perhaps a fire will be lit under someone's posterior to get it done. Godspeed to you guys.

[ShiftPlusOne]

I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

[kozman]

I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.

[dariuszb]

I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.

From my tests I can see that it is not per se ext4 issue but memory management of slab objects storing caches. All memory caches are affected I think, just ones related to ext4 operations are the easiest to fill in.

Having said that and anticipating that it might take some time to fix it wouldn't make sense to carry on with 4.14? The latest Pi release was 4.14.98 and it is already on 4.14.107

[kozman]

I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.

From my tests I can see that it is not per se ext4 issue but memory management of slab objects storing caches. All memory caches are affected I think, just ones related to ext4 operations are the easiest to fill in.

Having said that and anticipating that it might take some time to fix it wouldn't make sense to carry on with 4.14? The latest Pi release was 4.14.98 and it is already on 4.14.107

I'm certainly not smart enough to even understand the issue deeply or technically. I just troll through the RCs as they come to see what's coming down the pipe. From what I've read in the threads I posted, it *almost* sounds like a single component fix won't be the end all be all to fix the bug. I almost wonder if it'll be a combo fix of ext4 itself in tandem with the slab cache stuff. They are quite interwoven and interrelated that I would think significant changes in one would need some significant changes in the other. It's still interesting to see the gears turning in people heads on how to approach a fix. 50 different people will give you 50 different answers.

[dom]

Potential workaround for the inode slab leak is now available in rpi-update kernel.
Affected users, please update and report back.

The workaround disables memory and IO cgroups which appears to be the cause of the leaks.