Moving Linux kernel to 4.14

[verkerkbr]

I remeber seeing an update of Piclone about 5-6 months ago. Result is a better performance. Interesting is also that Piclone can copy 16 GB tot 32 GB and visa versa. You cannot do that with an image of your system with DD. Sometimes the image made is larger then the SD I like to use as a copy medium.

32GB is not always 32GB !

Regards,

Bram

[verkerkbr]

Latest kernel update to 4.14.69-V7+ is now running for several hours in combination with the marine chartplotter software OpenCPN.. No more lock-ups. Also tried Piclone, does work correctly.

Problems with leaked memory ? Seems be solved.

All is running with the VC4 driver on.

Thanks for the efforts. Systems are now working well and fast with the latest kernel update.

Regards,

Bram

[kozman]

Latest kernel update to 4.14.69-V7+ is now running for several hours in combination with the marine chartplotter software OpenCPN.. No more lock-ups. Also tried Piclone, does work correctly.

Problems with leaked memory ? Seems be solved.

All is running with the VC4 driver on.

Thanks for the efforts. Systems are now working well and fast with the latest kernel update.

Regards,

Bram

Good to hear. Must have just been a kernel bug. 4.14.70 was pushed out but if things are working well for you, probably no need to upgrade.

[verkerkbr]

Navigating chart ploitter software now running for several days on the RPI 3B+. With the latest kernel 4.14.70 it is working great.

Piclone also works without problems.

Bram

[kozman]

Navigating chart ploitter software now running for several days on the RPI 3B+. With the latest kernel 4.14.70 it is working great.

Piclone also works without problems.

Bram

Well that's good to know. I did a rpi-update a couple days ago after already being on 4.14.70 myself and some updates installed alongside some other fixes. Haven't seen any crashes or garbage on reboot since. You can be our guinea pig and try the newly minted 4.14.71.

[verkerkbr]

I'am sorry to inform that the kernel 4.14.71 gives OpenGL screen errors with zooming in-out with my Marine Navigation system. After some movements on the screen the system locks-up.

That is not the case with kernel version 4.14.70 and the other upgrades

I'am now back to 4.14.70. This version is working well.

Regards,

Bram

[verkerkbr]

Did some more trials with kernel .71. It is working fast and without problems in the default window setting of my Chartplotter navigation system. with OpenGL on.

The before reported errors in .71 appear only If I change the window frame to another format. Done by pulling at the window frame corner.

Bram

[kozman]

Did some more trials with kernel .71. It is working fast and without problems in the default window setting of my Chartplotter navigation system. with OpenGL on.

The before reported errors in .71 appear only If I change the window frame to another format. Done by pulling at the window frame corner.

Bram

Hm, makes me wonder if the desktop manager has a bug then. I upped to .71 on my RetroPi setup and nothing broke OpenGL fo rme. Stable so far.

[underwhelmd]

I had trouble at the beginning of the summer with whatever kernel version was current at that time. It happened on all the pi at my house which use pi cameras. CPU use would go through the roof. After reverting to a previous kernel, problem went away. Saw no reason to update since.

Decided to upgrade to 4.14.70 on one pi 3 yesterday. So far so good. I will leave it running for a few days before updating the rest but the previous problem would appear right away... has not happened. No errors.

Wanted to thank the people who develop this. Cheers.

[kozman]

So although it hasn't been made official, talk of what the next kernel will be has hovered around 4.19. A lot of good support and perf improvements have been added for RPi since kernel 4.14. While some upstream stuff has been back-ported to 4.14 kernel, I am curious if the devs ever considered, if possible, leveraging some of the perf / optimizations being used in Clear Linux?

As a regular reader of Phoronix, I regularly see Clear Linux beating the pants off of many popular distros (https://www.phoronix.com/scan.php?page= ... ling&num=2), (https://www.phoronix.com/scan.php?page= ... inux&num=2). An article discussing how these optimizations are tuned for Intel's architecture but can be extended to other platforms was talked about here: https://www.phoronix.com/scan.php?page= ... erformance

I know that GCC 6.4 was where ARM got a proverbial shot in the arm for perf improvements, but surely there's more blood to be squeezed from the stone? Debian Buster looks like it will be very good for the Pi once it's released next year.

[verkerkbr]

Just tried the kernel 4.14.72 and can report that it works faster again and now without video problems.

All is running very well. What a progress since the early days of the RPI. The software seems more important then the hardware in the performance of the RPI.

Thanks for all the efforts.

Bram

[kozman]

Just tried the kernel 4.14.72 and can report that it works faster again and now without video problems.

All is running very well. What a progress since the early days of the RPI. The software seems more important then the hardware in the performance of the RPI.

Thanks for all the efforts.

Bram

All good news! I too am finding the .70 series to be more performant on my RetroPi. Haven't bumped to .72 but .73 is already baking. No ARM fixes that I can see but some work for VC4 (Unity related, it seems) is. Might want to let someone else be the guinea pig!

[verkerkbr]

Now I have been using the latest kernel 4.14.72 plus OpenGL VC4 driver on and OpenGL setting in my marine chart plotter OpenCPN also on.

But this time for a longer test on board.

Works perfect with very fast OpenGL screen movements. System is now running more then 48 hours with GPS/AIS connected and the nauitical charts on the screen , without any problems or lock-ups. Great !

Regards,

Bram

[kozman]

Now I have been using the latest kernel 4.14.72 plus OpenGL VC4 driver on and OpenGL setting in my marine chart plotter OpenCPN also on.

But this time for a longer test on board.

Works perfect with very fast OpenGL screen movements. System is now running more then 48 hours with GPS/AIS connected and the nauitical charts on the screen , without any problems or lock-ups. Great !

Regards,

Bram

Just curious but are you using current git of the VC4 or the optional install? I was reading a few weeks ago about Eric H's VC4 work and it sound like a good perf boost was recently realized (https://anholt.github.io/twivc4/2018/09/07/twiv/). Now if only we could have a close to final driver to use.

[JamesPi123]

Does Raspberry Pi only use LTS kernels e.g. 4.9 4.14?

[6by9]

Does Raspberry Pi only use LTS kernels e.g. 4.9 4.14?

The main supported kernel will always be an lts kernel. The other kernel versions are in the github repo and will be brought up (mainly to keep the downstream patches current), but support is limited and will cease when the next version comes out.

[verkerkbr]

Re: Kozman,

I'am using the optional VC4 driver on the RPI 3b+ and now with the 4.14.73 kernel, the OpenCPNsoftware is running faster again.

I have read the info of Eric Anholt of the VC4 OpenGL driver, but have no idea how to get the latest VC4 driver.

Regards,


Bram

[kozman]

Re: Kozman,

I'am using the optional VC4 driver on the RPI 3b+ and now with the 4.14.73 kernel, the OpenCPNsoftware is running faster again.

I have read the info of Eric Anholt of the VC4 OpenGL driver, but have no idea how to get the latest VC4 driver.

Regards,


Bram

You'd likely have to get the VC4 source from latest git and compile yourself. I know Eric has been working on it for a *long* time but I feel like I read a while back that towards the end of this year he was hoping to have something for public consumption. I'm sure with his new baby that might not happen. Still, good to hear that .73 kernel is doing better than .72 and before. This is good news for others using the VC4 driver! \o/

[verkerkbr]

Perhaps the VC4 driver is improved, but I cannot find the source of for it.

After some longer use as a marine navigation chart plotter. The present OpenGL VC4 driver in combination with the latest kernel 4.14.73 works perfect with fast screen movements.

RPI3 B+ is running on OpenPlotter software based on Debian Stretch. Running GPS and AIS (Automatic Indification system for shipping) Navigation system is OpenCPN. Has Wifi Access point , VNC server etc. etc.

What an improvement in the software for the RPI !

[kozman]

Perhaps the VC4 driver is improved, but I cannot find the source of for it.

After some longer use as a marine navigation chart plotter. The present OpenGL VC4 driver in combination with the latest kernel 4.14.73 works perfect with fast screen movements.

RPI3 B+ is running on OpenPlotter software based on Debian Stretch. Running GPS and AIS (Automatic Indification system for shipping) Navigation system is OpenCPN. Has Wifi Access point , VNC server etc. etc.

What an improvement in the software for the RPI !

Looks like 4.14.74 was just pushed 3 hours ago. No VC4 fixes but some ARM/ARM64 fixes were.

[DougieLawson]

Is anyone else seeing these kernel ooops with 4.14.74+

Oct 8 2018 17:47:48
Copyright (c) 2012 Broadcom
version 656741eb5ba785fc4f1014a3a3b1c0e9c2cc8487 (clean) (release)

.firmware_revision e0a0ba809ef11f6bed8a00d34dda961239ea432c

Code: Select all

Oct 14 03:07:04 odyssey kernel: [96273.989232] enxb827eb355259: hw csum failure
Oct 14 03:07:04 odyssey kernel: [96273.989265] CPU: 0 PID: 0 Comm: swapper Tainted: G         C      4.14.74+ #1149
Oct 14 03:07:04 odyssey kernel: [96273.989272] Hardware name: BCM2835
Oct 14 03:07:04 odyssey kernel: [96273.989342] [<c0016508>] (unwind_backtrace) from [<c0013e40>] (show_stack+0x20/0x24)
Oct 14 03:07:04 odyssey kernel: [96273.989375] [<c0013e40>] (show_stack) from [<c0634424>] (dump_stack+0x20/0x28)
Oct 14 03:07:04 odyssey kernel: [96273.989404] [<c0634424>] (dump_stack) from [<c0540fe0>] (netdev_rx_csum_fault+0x3c/0x48)
Oct 14 03:07:04 odyssey kernel: [96273.989436] [<c0540fe0>] (netdev_rx_csum_fault) from [<c0534938>] (__skb_checksum_complete+0xc0/0xc4)
Oct 14 03:07:04 odyssey kernel: [96273.989474] [<c0534938>] (__skb_checksum_complete) from [<c05bc130>] (__udp4_lib_rcv+0x140/0x8cc)
Oct 14 03:07:04 odyssey kernel: [96273.989497] [<c05bc130>] (__udp4_lib_rcv) from [<c05bccb4>] (udp_rcv+0x20/0x28)
Oct 14 03:07:04 odyssey kernel: [96273.989532] [<c05bccb4>] (udp_rcv) from [<c0587eb8>] (ip_local_deliver_finish+0xd0/0x2a8)
Oct 14 03:07:04 odyssey kernel: [96273.989558] [<c0587eb8>] (ip_local_deliver_finish) from [<c0588670>] (ip_local_deliver+0x54/0xdc)
Oct 14 03:07:04 odyssey kernel: [96273.989579] [<c0588670>] (ip_local_deliver) from [<c05882e8>] (ip_rcv_finish+0x258/0x454)
Oct 14 03:07:04 odyssey kernel: [96273.989598] [<c05882e8>] (ip_rcv_finish) from [<c0588970>] (ip_rcv+0x278/0x3fc)
Oct 14 03:07:04 odyssey kernel: [96273.989617] [<c0588970>] (ip_rcv) from [<c053ee28>] (__netif_receive_skb_core+0x7c0/0xcc8)
Oct 14 03:07:04 odyssey kernel: [96273.989637] [<c053ee28>] (__netif_receive_skb_core) from [<c0541138>] (__netif_receive_skb+0x20/0x7c)
Oct 14 03:07:04 odyssey kernel: [96273.989655] [<c0541138>] (__netif_receive_skb) from [<c0541214>] (process_backlog+0x80/0x114)
Oct 14 03:07:04 odyssey kernel: [96273.989678] [<c0541214>] (process_backlog) from [<c0544be8>] (net_rx_action+0x240/0x380)
Oct 14 03:07:04 odyssey kernel: [96273.989698] [<c0544be8>] (net_rx_action) from [<c00095c4>] (__do_softirq+0x124/0x328)
Oct 14 03:07:04 odyssey kernel: [96273.989728] [<c00095c4>] (__do_softirq) from [<c0026764>] (irq_exit+0xf0/0x140)
Oct 14 03:07:04 odyssey kernel: [96273.989765] [<c0026764>] (irq_exit) from [<c00619f8>] (__handle_domain_irq+0x60/0xb8)
Oct 14 03:07:04 odyssey kernel: [96273.989786] [<c00619f8>] (__handle_domain_irq) from [<c0009418>] (bcm2835_handle_irq+0x28/0x48)
Oct 14 03:07:04 odyssey kernel: [96273.989809] [<c0009418>] (bcm2835_handle_irq) from [<c064e31c>] (__irq_svc+0x5c/0x7c)
Oct 14 03:07:04 odyssey kernel: [96273.989819] Exception stack(0xc093def8 to 0xc093df40)
Oct 14 03:07:04 odyssey kernel: [96273.989830] dee0:                                                       00000000 00000000
Oct 14 03:07:04 odyssey kernel: [96273.989847] df00: ffffffff c093f414 c093c000 c093f0a4 c09c7906 c093f020 c09d4920 c0917a28
Oct 14 03:07:04 odyssey kernel: [96273.989863] df20: d7fffa20 c093df54 c093df48 c093df48 c001079c c00107a0 60000013 ffffffff
Oct 14 03:07:04 odyssey kernel: [96273.989888] [<c064e31c>] (__irq_svc) from [<c00107a0>] (arch_cpu_idle+0x30/0x40)
Oct 14 03:07:04 odyssey kernel: [96273.989908] [<c00107a0>] (arch_cpu_idle) from [<c064e1b4>] (default_idle_call+0x34/0x48)
Oct 14 03:07:04 odyssey kernel: [96273.989926] [<c064e1b4>] (default_idle_call) from [<c0053db8>] (do_idle+0x8c/0xec)
Oct 14 03:07:04 odyssey kernel: [96273.989999] [<c0053db8>] (do_idle) from [<c0054094>] (cpu_startup_entry+0x1c/0x20)
Oct 14 03:07:04 odyssey kernel: [96273.990034] [<c0054094>] (cpu_startup_entry) from [<c0648bfc>] (rest_init+0x7c/0x9c)
Oct 14 03:07:04 odyssey kernel: [96273.990063] [<c0648bfc>] (rest_init) from [<c08ced48>] (start_kernel+0x358/0x3c8)
Oct 14 03:07:07 odyssey kernel: [96276.533200] enxb827eb355259: hw csum failure

On RPi 1B & RPi 1B+ that have a wired interface?

It's disappeared on a reboot (for the moment).

[rasporg]

Seeing this also on a 3B+ running on Jessie after a firmware update.

Code: Select all

WARNING:  Kernel Errors Present
    [<7f193454>] (tcp_error [nf_conntrack])  ...:  11 Time(s)
    [<807331c8>] (nf_ip_checksum) from [<7f193454>] (tcp_error+0x1d0/0x21c [nf_ ...:  11 Time(s)

1 Time(s): 1ee0:                                     00000000 621bdf0c 397c2000 00000000
1 Time(s): 1ee0:                                     00000000 621bfa7c 397c2000 00000000
1 Time(s): 1ee0:                                     00000000 621c3f34 397c2000 00000000
1 Time(s): 1ee0:                                     00000000 621f8f88 397c2000 00000000
1 Time(s): 1ee0:                                     00000000 621f90a0 397c2000 00000000
4 Time(s): 1ee0:                                     00000000 621f9dfc 397c2000 00000000
1 Time(s): 1ee0:                                     00000000 6af26b80 397c2000 00000000
10 Time(s): 1f00: 80c00000 80c03dcc 80c03d68 80c885b2 00000001 80b60a30 babff9c0 80c01f4c
9 Time(s): 1f20: 80c04174 80c01f40 80108a4c 80108a50 60000013 ffffffff
1 Time(s): 1f20: 80c04174 80c01f40 80108a4c 80108a50 600f0013 ffffffff
1 Time(s): 9fa0:                                     00000000 00000000 015d0b60 00000000
1 Time(s): 9fc0: 015b8150 00000000 01656050 00000000 00000000 00000000 005e9000 015b8150
1 Time(s): 9fe0: 00000000 7e98d520 004cae35 004c511c 60010030 ffffffff
10 Time(s): CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C      4.14.74-v7+ #1149
1 Time(s): CPU: 0 PID: 21591 Comm: monitor.pl Tainted: G         C      4.14.74-v7+ #1149
10 Time(s): Exception stack(0x80c01ef0 to 0x80c01f38)
1 Time(s): Exception stack(0xa0e59fb0 to 0xa0e59ff8)
11 Time(s): Hardware name: BCM2835
11 Time(s): [<7f18d994>] (nf_conntrack_in [nf_conntrack]) from [<7f1df2dc>] (ipv4_conntrack_in+0x28/0x2c [nf_conntrack_ipv4])
11 Time(s): [<7f1df2dc>] (ipv4_conntrack_in [nf_conntrack_ipv4]) from [<806d04dc>] (nf_hook_slow+0x4c/0xd0)
10 Time(s): [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<807a41bc>] (__irq_svc+0x5c/0x7c)
1 Time(s): [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<807a450c>] (__irq_usr+0x4c/0x60)
11 Time(s): [<80101694>] (__do_softirq) from [<80123870>] (irq_exit+0x108/0x164)
10 Time(s): [<80108a50>] (arch_cpu_idle) from [<807a393c>] (default_idle_call+0x34/0x48)
11 Time(s): [<8010c240>] (show_stack) from [<80788604>] (dump_stack+0xd4/0x118)
11 Time(s): [<8010ffd4>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
11 Time(s): [<80123870>] (irq_exit) from [<801759a8>] (__handle_domain_irq+0x70/0xc4)
10 Time(s): [<801614b8>] (do_idle) from [<801617cc>] (cpu_startup_entry+0x28/0x2c)
10 Time(s): [<801617cc>] (cpu_startup_entry) from [<8079d664>] (rest_init+0xbc/0xc0)
11 Time(s): [<801759a8>] (__handle_domain_irq) from [<80101504>] (bcm2836_arm_irqchip_handle_irq+0xa8/0xac)
11 Time(s): [<80681144>] (__skb_checksum_complete) from [<807331c8>] (nf_ip_checksum+0xd4/0x130)
11 Time(s): [<8068be4c>] (__netif_receive_skb_core) from [<8068e9e4>] (__netif_receive_skb+0x20/0x7c)
11 Time(s): [<8068e870>] (netdev_rx_csum_fault) from [<80681144>] (__skb_checksum_complete+0xbc/0xc0)
11 Time(s): [<8068e9e4>] (__netif_receive_skb) from [<8068ead8>] (process_backlog+0x98/0x148)
11 Time(s): [<8068ead8>] (process_backlog) from [<80692df0>] (net_rx_action+0x2e8/0x45c)
11 Time(s): [<80692df0>] (net_rx_action) from [<80101694>] (__do_softirq+0x18c/0x3d8)
11 Time(s): [<806d04dc>] (nf_hook_slow) from [<806d8df4>] (ip_rcv+0x460/0x514)
11 Time(s): [<806d8df4>] (ip_rcv) from [<8068be4c>] (__netif_receive_skb_core+0x340/0xc84)
11 Time(s): [<80788604>] (dump_stack) from [<8068e870>] (netdev_rx_csum_fault+0x44/0x48)
10 Time(s): [<8079d664>] (rest_init) from [<80b00df8>] (start_kernel+0x3d4/0x3e0)
10 Time(s): [<807a393c>] (default_idle_call) from [<801614b8>] (do_idle+0xd8/0x150)
10 Time(s): [<807a41bc>] (__irq_svc) from [<80108a50>] (arch_cpu_idle+0x34/0x4c)
11 Time(s): eth0: hw csum failure

[jamesh]

Hmm, another checksum issue. There are a couple of github issues that may be relevant.

[DougieLawson]

Hmm, another checksum issue. There are a couple of github issues that may be relevant.

I found one - that's new enough and relevant. I can't work out what https://github.com/Hexxeh commit I need to go back to to avoid the error.

[6by9]

Hmm, another checksum issue. There are a couple of github issues that may be relevant.

I found one - that's new enough and relevant. I can't work out what https://github.com/Hexxeh commit I need to go back to to avoid the error.

You commented on the issue which included

tried every single version from that - seems to happen from c919d63

The issue also references https://github.com/raspberrypi/linux/issues/2659 which includes

From #2713, it appears that issues on 4.14 have occurred at 4.14.71.

Could those seeing issues on 4.14 try reverting to 4.14.70 (sudo rpi-update e880e627d11af2b17be3efb6d105e3c28cd63867) and see if they still get the issues. Moving forward the one step to 4.14.71 (sudo rpi-update c919d632ddc2a88bcb87b7d0cddd61446d1a36bf) is expected to show up the problems.

For 4.18 it isn't so easy, but having looked at the commits between 4.14.70 - 71 I'd be suspecting 88078d9, so if someone could try 4.18 with that commit reverted then it'd be a useful test.

It's looking like it's an upstream commit that is causing the issue. We'll be looking at it tomorrow, and probably reverting it in our repo whilst it is reported and investigated upstream.