dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Tue Mar 26, 2019 4:12 pm

dom wrote:
Tue Mar 26, 2019 2:22 pm
Potential workaround for the inode slab leak is now available in rpi-update kernel.
Affected users, please update and report back.

The workaround disables memory and IO cgroups which appears to be the cause of the leaks.
Good news! Sure I will try in spare time. In the meantime I have a question.

What are we missing with disabled memory and IO cgroups? I am not an expert on linux kernel but obviously it is on for some reason. Understanding this I can check not only that system does not crash in situations like before but also what else might be impacted. I did quick wikipedia on cgroups but I am not clear at all what I can expect with the system without it vs with it.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Mar 26, 2019 4:25 pm

Mostly a scheme for isolating resource usage for collections of processes.
Mostly used for virtualisation, or containerisation (like docker).

I don't think 99% of users will notice its absence, but this is what the test builds are for.
If things stop working due to its absence, then report it here.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Tue Mar 26, 2019 4:31 pm

dom wrote:
Tue Mar 26, 2019 4:25 pm
Mostly a scheme for isolating resource usage for collections of processes.
Mostly used for virtualisation, or containerisation (like docker).

I don't think 99% of users will notice its absence, but this is what the test builds are for.
If things stop working due to its absence, then report it here.
Cool! So looks like not a big deal for small RPi projects I run. I will dip my toe into it and let you guys know.

teemu.niiranen
Posts: 2
Joined: Wed Mar 27, 2019 2:57 pm

Re: Moving Linux kernel to 4.19

Wed Mar 27, 2019 2:59 pm

For AWS Greengrass 4.19 is a no go. No way to setup max memory for local Lambda functions?

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Wed Mar 27, 2019 3:33 pm

teemu.niiranen wrote:
Wed Mar 27, 2019 2:59 pm
For AWS Greengrass 4.19 is a no go. No way to setup max memory for local Lambda functions?
I think you'll have to explain more precisely. Are you referring to the lack of cgroups, the slab/inode leak or some other change in 4.19.

lems
Posts: 2
Joined: Thu Mar 07, 2019 7:14 am

Re: Moving Linux kernel to 4.19

Wed Mar 27, 2019 5:55 pm

Seems to work fine now. The rsnapshot/rsync command that failed previously completed succesfully now running the latest rpi-kernel update (4.19.30; 00bb2894c0a4349464c19ba62f62b8d9e17a1a9e).

Thank you!

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Wed Mar 27, 2019 7:00 pm

lems wrote:
Wed Mar 27, 2019 5:55 pm
Seems to work fine now. The rsnapshot/rsync command that failed previously completed succesfully now running the latest rpi-kernel update (4.19.30; 00bb2894c0a4349464c19ba62f62b8d9e17a1a9e).
Thanks for confirming.

teemu.niiranen
Posts: 2
Joined: Wed Mar 27, 2019 2:57 pm

Re: Moving Linux kernel to 4.19

Wed Mar 27, 2019 7:40 pm

dom wrote:
Wed Mar 27, 2019 3:33 pm
teemu.niiranen wrote:
Wed Mar 27, 2019 2:59 pm
For AWS Greengrass 4.19 is a no go. No way to setup max memory for local Lambda functions?
I think you'll have to explain more precisely. Are you referring to the lack of cgroups, the slab/inode leak or some other change in 4.19.
Lack of cgroups is the case.

"Edit your command line boot file to enable and mount memory cgroups. This allows AWS IoT Greengrass to set the memory limit for Lambda functions. Without this, the Greengrass daemon is unable to run."
- Source: https://docs.aws.amazon.com/greengrass/ ... dule1.html

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Thu Mar 28, 2019 12:01 pm

dom wrote:
Tue Mar 26, 2019 2:22 pm
Potential workaround for the inode slab leak is now available in rpi-update kernel.
Affected users, please update and report back.

The workaround disables memory and IO cgroups which appears to be the cause of the leaks.
I tested all cases causing issues previously with 4.19 and now they all pass. So looks like this workaround works. It is reasonably stable for me now so I switch all my RPis to it and will keep an eye on it.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Thu Mar 28, 2019 2:44 pm

teemu.niiranen wrote:
Wed Mar 27, 2019 7:40 pm
Lack of cgroups is the case.

"Edit your command line boot file to enable and mount memory cgroups. This allows AWS IoT Greengrass to set the memory limit for Lambda functions. Without this, the Greengrass daemon is unable to run."
- Source: https://docs.aws.amazon.com/greengrass/ ... dule1.html
Okay, best to avoid 4.19 for now. We'll enable cgroups again when the upstream issues are resolved.
But for testing now, disabling cgroups is the lesser of two evils.

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Moving Linux kernel to 4.19

Fri Mar 29, 2019 9:45 pm

Man I´ve gone like crazy (like the Pi 3 B+ did the last months)! Spent sooooo much time on this, really DAYS, coming to conclusion: it must be a memory leak and very likely something in the kernel. rpi-update finally brought me here, seeing others suffering the same.

So: Finally Kernel v4.19.30 seems to have fixed it! That´s what it looks like at the moment. Cause at least rsync backup tasks can finally finish without RC12 again cause no memory bandit anymore which triggers OOM_KILLER and kills rsync. I´ll now monitor the system performance and stability closely for the next two to three weeks to give a final summary. Finger´s crossed.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Thu Apr 04, 2019 2:20 pm

I was just poking around 4.19.34 and there is some cgroup work present. Could it be? At long last....something [1]?

[1] https://lkml.org/lkml/2019/3/27/1315

User avatar
Gavinmc42
Posts: 3925
Joined: Wed Aug 28, 2013 3:31 am

Re: Moving Linux kernel to 4.19

Fri Apr 05, 2019 2:01 am

Does this mean it is getting close to pushing the "Go " button for using 4.19?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
DougieLawson
Posts: 36327
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Moving Linux kernel to 4.19

Fri Apr 05, 2019 6:37 am

Gavinmc42 wrote:
Fri Apr 05, 2019 2:01 am
Does this mean it is getting close to pushing the "Go " button for using 4.19?
Already half pushed. The mainstream kernel is being built at 4.19. The next apt update of raspberrypi-kernel will deliver it to everyone. 4.14 has gone dormant.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

anberra
Posts: 1
Joined: Tue Apr 09, 2019 4:00 pm

Re: Moving Linux kernel to 4.19

Tue Apr 09, 2019 4:04 pm

Hi,

I recently updated kernel to 4.19.34-v7+ #1211, and memory cgroups does not work anymore. It is important to bring my kubernetes cluster back to life.

Do you have any workaround?

Thank you!

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Tue Apr 09, 2019 6:54 pm

anberra wrote:
Tue Apr 09, 2019 4:04 pm
Hi,

I recently updated kernel to 4.19.34-v7+ #1211, and memory cgroups does not work anymore. It is important to bring my kubernetes cluster back to life.

Do you have any workaround?

Thank you!

Not as of yet. I know there was some cgroup work done in the 4.19.34 kernel so it obviously had a negative impact in your case.

You might want to read through https://cdn.kernel.org/pub/linux/kernel ... og-4.19.34 and see if you can reach out the to commit authors who made the changes and ask for advice.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Apr 09, 2019 7:56 pm

kozman wrote:
Tue Apr 09, 2019 6:54 pm
Not as of yet. I know there was some cgroup work done in the 4.19.34 kernel so it obviously had a negative impact in your case.
cgroups was disabled to avoid the memory leak (reported here).
We are waiting for a fix for the memory leak before we can enable cgroups again.

stuartiannaylor

Re: Moving Linux kernel to 4.19

Fri Apr 12, 2019 7:01 pm

I have been running the cvt2f2fs script from the viewtopic.php?p=1285848#p1285848 post by RonR which pulls 4.19 which is actually great as f2fs got a lot of commits in 4.18 and some in 4.19.

I guess I will try a few things and see how things go, just one question as this confuses me and haven't found anything to confirm.
mkfs.f2fs -l label /dev/block_device creates a f2fs filesystem and its inbuilt discard the default of on.
Confused as to if fstab/mount should have -o discard or not, discard with f2fs is inherent in the file system and been wondering that really it should not be a fstab/mount directive as its enabled at a filesystem level?

Systemd seems to be trying to run with an invalid option
fsck.f2fs: invalid option -- 'y'

I will give it a whirl though as will need to check if with root it needs some of the initcpio modules loaded that seem to be mentioned.
I will post back with fixes or if I just hit a cul-de-sac
Nope just me being a dope as needed to do the usual apt-get install -t buster setup to get f2fs-tools from the buster repo and all seems well.
There is a boot delay added prob now that fsck.f2fs is working prob takes 3-5 secs.

Code: Select all

Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: sector size = 512
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: total sectors = 30128128 (14711 MB)
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: MKFS version
Apr 12 23:00:43 raspberrypi systemd-fsck[130]:   "Linux version 4.19.34-v7+ ([email protected]) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1211 SMP Mon Apr 8 22:56:37 BST 2019"
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: FSCK version
Apr 12 23:00:43 raspberrypi systemd-fsck[130]:   from "Linux version 4.19.34-v7+ ([email protected]) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1211 SMP Mon Apr 8 22:56:37 BST 2019"
Apr 12 23:00:43 raspberrypi systemd-fsck[130]:     to "Linux version 4.19.34-v7+ ([email protected]) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1211 SMP Mon Apr 8 22:56:37 BST 2019"
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: superblock features = 0 :
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: total FS sectors = 30128128 (14711 MB)
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: CKPT version = 157
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Info: checkpoint state = 45 :  crc compacted_summary unmount
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] Unreachable nat entries                        [Ok..] [0x0]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] SIT valid block bitmap checking                [Ok..]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] Hard link checking for regular file            [Ok..] [0x0]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] valid_block_count matching with CP             [Ok..] [0x5b061]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] valid_node_count matcing with CP (de lookup)   [Ok..] [0xb769]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0xb769]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] valid_inode_count matched with CP              [Ok..] [0xb6f0]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] free segment_count matched with CP             [Ok..] [0x19a0]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] next block offset is free                      [Ok..]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] fixing SIT types
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: [FSCK] other corrupted bugs                           [Ok..]
Apr 12 23:00:43 raspberrypi systemd-fsck[130]: Done.
Thinks something to do with the hwclock and network time

Code: Select all

Apr 12 23:00:43 raspberrypi kernel: [    0.310838] bcm2835-rng 3f104000.rng: hwrng registered
Apr 12 23:00:43 raspberrypi systemd-udevd[191]: Process '/usr/sbin/th-cmd --socket /var/run/thd.socket --passfd --udev' failed with exit code 1.
Apr 12 23:00:43 raspberrypi systemd-udevd[186]: Process '/usr/sbin/th-cmd --socket /var/run/thd.socket --passfd --udev' failed with exit code 1.
Apr 12 23:00:43 raspberrypi systemd-udevd[202]: Process '/usr/sbin/th-cmd --socket /var/run/thd.socket --passfd --udev' failed with exit code 1.
Apr 12 23:00:43 raspberrypi systemd-udevd[189]: Process '/usr/sbin/th-cmd --socket /var/run/thd.socket --passfd --udev' failed with exit code 1.
Apr 12 23:00:43 raspberrypi systemd[1]: Started Network Time Synchronization.
I think actually the above is just something I never noticed before and its just fsck.f2fs gives a 5-10 sec delay and whatever it is with triggerhappy guess will have to check on a clean install if it was there as triggerhappy is already the newest version (0.5.0-1) -t buster

Yeap as disable in fstab and boot back to normal /dev/mmcblk0p2 / f2fs defaults,noatime,discard 0 0
Dunno even sudo shutdown -f now if fstab is enabled means it runs everytime.
Did a apt-get install -t buster e2fsprogs
But still
tune2fs: Bad magic number in super-block while trying to open /dev/mmcblk0p2
/dev/mmcblk0p2 contains a f2fs file system
Think it is my cul-de-sac as guess will just have to wait for Buster and check why the script runs rpi-update as not really sure what the tune2fs for f2fs is as was just going to change to chk every n boots.

If you edit the script and delete the rpi-update and stick with original kernel fsck.f2fs doesn't run on each boot.
But guess with fingers crossed it might only be months before Buster and 4.18 did get a lot of commits for f2fs.
I was just checking that script didn't even know it would bump the kernel but something somewhere is not the same as it is for Stretch.

Nope in stretch it does the same and the f2fs-tools and kernel seem to be out of sync as getting systemd-fsck[117]: fsck.f2fs: invalid option -- 'y' once more without rpi-update.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Thu Apr 25, 2019 2:21 pm

cgroups is back in with latest rpi-update kernel and hopefully memory leak is still fixed.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Thu Apr 25, 2019 4:22 pm

dom wrote:
Thu Apr 25, 2019 2:21 pm
cgroups is back in with latest rpi-update kernel and hopefully memory leak is still fixed.
Great news! Have upgraded one RPi and initial tests look promising. Memory leak is not anymore. I will keep an eye on this and let it run for longer doing some real life jobs.

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Thu Apr 25, 2019 4:27 pm

dom wrote:
Thu Apr 25, 2019 2:21 pm
cgroups is back in with latest rpi-update kernel and hopefully memory leak is still fixed.
Yay! Just curious, was it due to fixes in .35 / .36 or a fix pulled from upstream? Nothing in .35 / .36 jumped out at me.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Thu Apr 25, 2019 4:33 pm

kozman wrote:
Thu Apr 25, 2019 4:27 pm
Yay! Just curious, was it due to fixes in .35 & .36 or or a fix pulled from upstream? Nothing in .35/.36 jumped out at me.
Looks like it was self inflicted - in the past in order to save memory Pi specific patch was applied.

https://github.com/raspberrypi/linux/co ... 030263149b

All worked well until 4.19

reverting it solved the issue

https://github.com/raspberrypi/linux/pull/2941

I wonder what is penalty? meaning what sort of size of memory we are talking about

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5342
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Thu Apr 25, 2019 7:22 pm

dariuszb wrote:
Thu Apr 25, 2019 4:33 pm
I wonder what is penalty? meaning what sort of size of memory we are talking about
I believe it's 8 bytes per 4K page of the system is lost up front, so 2MB less memory on boot.

(when first introduced we also had CGROUP_MEM_RES_CTLR enabled as Epiphany made use of that,
and that adds an extra 20 bytes per 4K page, so an additional 5MB lost. I think this setting no longer exists in newer kernels).

rkk2025
Posts: 3
Joined: Tue Apr 04, 2017 6:19 pm

Re: Moving Linux kernel to 4.19

Fri Apr 26, 2019 11:52 pm

Is there any way to get the kernel headers for the latest 4.19 kernel version? (To be used with DKMS)

User avatar
DougieLawson
Posts: 36327
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Moving Linux kernel to 4.19

Sat Apr 27, 2019 1:00 am

Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Return to “Advanced users”