Pi3 memory leak?

[MartyMacGyver]

I got my Pi3 soon after it was released. I cloned my Raspbian install from my Pi2, renamed the device, booted it and updated it appropriately (including undoing overclocking things not appropriate to the Pi3).

The Pi2 and the Pi3 are both online and are basically identical. However, while the Pi2 will run for weeks at a time (25 days uptime since my last reboot), the Pi3 will drop offline after a few days. I thought it was the WiFi dropping out, causing it to become inaccessible (despite a small WiFi restart script I successfully have running on both devices). However, I don't think that's the problem.

According to 'top', the Pi2 uses a fair amount of memory but that amount has remained stable:
KiB Mem: 883052 total, 401412 used, 481640 free, 108220 buffers
(htop claims 91/862 MB are in use - likewise, stable)

However, the Pi3 memory usage has steadily increased - it's been online just over a day and already:
KiB Mem: 947748 total, 888536 used, 59212 free, 81880 buffers
(htop says 607/925 MB are in use... this has risen by 4 MB in the short time I've been typing this post.)

I suspect a memory leak, but it's not clear what's doing it. Unlike the Pi2, I found the OOM task killer is being activated, shortly before the logs just *stop* (the app happened to be my WiFi shell script, but I believe it triggered it because it happens to run periodically and tried to run when memory was maxed out already - there's nothing prone to memory leakage in it). There is ample "disk" space on each device; swap is available (1023MB) but is currently unused.

I realize this isn't a pristine install, but I don't see why it should be leaking like this either. Thoughts on how I could track down what part of the system is slowly draining my memory? As I've seen other mentions of memory leaks (possibly in the WiFi subsystem itself) it would be useful *and* educational to track down the leak.

Edit: it's 10 minutes later and now swap is getting involved on the Pi3. It's just 1 MB but give it time...

[jamesh]

Have you tried a vanilla install of the latest Raspbian? DOes it happen on that?

[MaxK1]

What kernel are you using? 'uname -a will' give you that info. Does anything show up in 'dmesg'?

[MartyMacGyver]

No, I haven't tried vanilla because I want to learn how to debug the problem as it happens (to see what's leaking, if that's even possible).

The kernel is:
Linux raspberrypi-3a 4.4.10-v7+ #885 SMP Fri May 13 15:44:27 BST 2016 armv7l GNU/Linux

There is nothing helpful in the dmsg logs. The notable difference in the syslogs between the two machines is the frequent "brcmfmac: brcmf_sdio_hdparse: seq xx: sequence number error, expect yy" messages which are well-known to Pi3 users (it's not clear they'd have this effect).

[MaxK1]

Yep - The wi-fi driver is the problem I reported a few days ago. Seems that every time the system throws a "brcmfmac: brcmf_sdio_hdparse: seq xx: sequence number error, expect yy" , another 'n' (4?) M just vanishes... I'm using Ralink wi-fi dongles on both my Pi 3's - no problems with 4.4.10.

[MarkHaysHarris777]

Use htop to look carefully at each process... you should be able to see which process is gobbling up the memory... another thing you can do is close thread|processes one by one until the leak goes away; try a combination of the two (intelligently guess which process is the culprit and close it with f9 SIGTERM and see if you can contain the leak.

(I sympathize... I hate memory leaks, they are the worst)

If you can't track it down set a crontab entry that reboots the system at midnight until you get a handle on it.

[MaxK1]

Sorry Marcus - there is no 'process' that is chewing up the memory. It's the Broadcom driver. You can easily reproduce the problem by opening several tabs in iceweasel and refreshing them constantly.

[MartyMacGyver]

I mirrored this question on Reddit (https://redd.it/4jvm8s) and got a couple extra suggestions: lsmod and slabtop.

lsmod didn't show anything interesting (neither did top/htop).

slabtop certainly did though! On my Pi3:
341678 341678 100% 2.00K 39372 16 1259904K kmalloc-2048

There's nothing remotely like that going on on the Pi2. It may indeed be the WiFi driver - which worries me if it ends up being like some other long-standing bugs that never end up getting fixed (the I2C clock stretching problem springs unhappily to mind).

I'm curious what others see for slabtop on their Pi3s? (You may need to run as root, e.g., 'sudo slabtop')

[MartyMacGyver]

Yep - The wi-fi driver is the problem I reported a few days ago. Seems that every time the system throws a "brcmfmac: brcmf_sdio_hdparse: seq xx: sequence number error, expect yy" , another 'n' (4?) M just vanishes... I'm using Ralink wi-fi dongles on both my Pi 3's - no problems with 4.4.10.

So are you saying things are fine when you use an external WiFi adapter on the Pi3 (presumably disabling the internal one), but have similar problems with the internal one?

Is this your bug?
https://github.com/raspberrypi/linux/issues/1471

(I added my findings to that bug - hopefully it'll help get it resolved)

[MaxK1]

That isn't my bug report but it does decribe the problem.

I haven't seen it with the external dongle. Ralink...

[MartyMacGyver]

I was able to successfully build the kernel with kmemleak enabled - the output is interesting and is filled with the following (each pair of these leaks occurs about once per second):

Code: Select all

unreferenced object 0xa0c1abc0 (size 2048):
  comm "ifplugd", pid 647, jiffies 458066 (age 816.840s)
  hex dump (first 32 bytes):
    <...local routing info...>
  backtrace:
    [<801497ac>] kmem_cache_alloc_trace+0x1c4/0x2a0
    [<7f257a3c>] brcmf_cfg80211_get_station+0x29c/0x344 [brcmfmac]
    [<7f150cb8>] cfg80211_wireless_stats+0xa8/0x2b0 [cfg80211]
    [<805b0338>] get_wireless_stats+0x70/0x7c
    [<805b0364>] iw_handler_get_iwstats+0x20/0x94
    [<805b0100>] ioctl_standard_call+0x2f0/0x4b8
    [<805b05c0>] wext_handle_ioctl+0x1b8/0x23c
    [<804f02d4>] dev_ioctl+0x53c/0x800
    [<804b97c4>] sock_ioctl+0x12c/0x2b0
    [<8016c5e8>] do_vfs_ioctl+0x424/0x614
    [<8016c81c>] SyS_ioctl+0x44/0x6c
    [<8000fb40>] ret_fast_syscall+0x0/0x1c
    [<ffffffff>] 0xffffffff

Code: Select all

unreferenced object 0xa0c1c600 (size 2048):
  comm "ifplugd", pid 647, jiffies 458066 (age 816.840s)
  hex dump (first 32 bytes):
    <...local routing info...>
  backtrace:
    [<801497ac>] kmem_cache_alloc_trace+0x1c4/0x2a0
    [<7f257a3c>] brcmf_cfg80211_get_station+0x29c/0x344 [brcmfmac]
    [<7f150cb8>] cfg80211_wireless_stats+0xa8/0x2b0 [cfg80211]
    [<805b0338>] get_wireless_stats+0x70/0x7c
    [<805b07c0>] wireless_dev_seq_show+0x34/0x17c
    [<8017e5b8>] seq_read+0x3a0/0x4b8
    [<801bd5d0>] proc_reg_read+0x6c/0x94
    [<8015a31c>] __vfs_read+0x34/0xe0
    [<8015ab88>] vfs_read+0x8c/0x158
    [<8015b5b4>] SyS_read+0x54/0xb0
    [<8000fb40>] ret_fast_syscall+0x0/0x1c
    [<ffffffff>] 0xffffffff

At a leak rate of 4KB per pair, roughly every second, for a day, this is leaking about 350-400MB/day! This is very close to what I'm seeing in practice.

For reference, I've documented the full kernel build procedure including headers and (optionally) the kernel memory debug settings.

https://gist.github.com/MartyMacGyver/2 ... c1c9835adb

This is the bug that I think best tracks this issue:

https://github.com/raspberrypi/linux/issues/1471

[MaxK1]

Is there a USB dongle using the same Broadcom chipset and driver? It would be potentially interesting to see if the same problem exists with one of those as well...

[jamesh]

Check out the github thread - quite interesting. rpi-update should fix the issue.

[MartyMacGyver]

Indeed! I built from source and the leak is no longer happening (kmemleak confirms it). Unless there's some other periodic issue that wasn't evident because of this, my Pi should work indefinitely now (versus dying a couple days after booting it). I appreciate the fast turnaround from the Pi team!

[MaxK1]

That seems to have fixed it (although I still get loads of sequence errors; "brcmfmac: brcmf_sdio_hdparse: seq xx: sequence number error, expect yy") Good job! to everyone involved.