SurferTim
Posts: 1769
Joined: Sat Sep 14, 2013 9:27 am
Location: Miramar Beach, Florida

Re: Newest Kernel and / or Chromium broken

Sun Oct 07, 2018 12:40 pm

I usually don't criticize, but Shlendrian has a point. I'm having problems with Raspbian video/web browsers.

Just as a comparison, I have a few different types of small form factor boards. The Tinker Board, even tho it does not have the technical support as does the RPi, and is a bit pricier, outperforms the RPi hands down. I'm watching a 1080p/30fps full screen video streamed from YouTube smooth as silk. It also launches Chromium/widevide v67 with no problem, which allows Netflix and Amazon Prime. Try this video. It is awe-inspiring in 1080p.
https://youtu.be/6v2L2UGZJAM

This is a challenge to the RPi crew. Help me out here. Is there nothing that can be done to make your OS more Debian 9 compatible?

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 5:04 am

Schlendrian wrote:
Sun Oct 07, 2018 12:09 pm
This doesn't do anything :?: cups.service couldn't even be found (maybe, because I dont't have printer set up?).
That is correct, and the configuration options that tell processes like apt to automatically download and cache package lists can be a pain-in-the-network depending on your timezone and bandwidth access cost.
Schlendrian wrote:
Sun Oct 07, 2018 12:09 pm
I have a heat sink. Overheating is not the problem. The problem is, that things don't work with the newest updates, that worked flawlessly beforehand.
I would say a fan cooler is needed, if you are knocked back to using a software codec on some files.
Schlendrian wrote:
Sun Oct 07, 2018 12:09 pm
I understand, that maintaining a Kernel, Software and Firmware is very hard and stressful work. The problem is, that developers seem to try fixing problems, that aren't there. Things get destroyed for no reason. Why would you fix something, that doesn't need to be fixed? I mean, I'm not the only person complaining about all the issues, that came with the newest updates.
It is usually an inexperienced project managers ego that causes these kind of cultural shifts, but it is not really the Raspberry Pi foundations fault (except their blob driver and BSP layers). Debian has been going through some recent changes the past year, and it is mostly the non-optional non-posix philosophy of systemd dividing communities (I see both sides of the argument, and they both have trade-offs).
Senior software teams generally refer to the refactoring mess as "Yak shaving" ... and forks can be painful. ;-)
Schlendrian wrote:
Sun Oct 07, 2018 12:09 pm
See, I'm not here to harm anyone. Criticizing mistakes should be allowed. If people get hurt by critique and stop responding, it's not my fault.
I agree the release seemed rushed, but I think most people wanted to please impatient users like me awaiting their next gadget... and I have faith in their good intentions. =) In the future, people will maybe consider looking at a transitional support period where a tested "stable" release is preferred for awhile....

Identifying a technical issues solution often is a cooperative endeavor, as individuals we will miss what others see from a different vantage point. Being diligent about regression testing is unfortunately the package maintainer(s) responsibility, the Pi is still a proprietary API, and sometimes people abandon projects along with the bug reports (on-vacation/busy/distracted/dead/don't-care-anymore).

Cheers,
~J~

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 9:51 am

SurferTim wrote:
Sun Oct 07, 2018 12:40 pm
I usually don't criticize, but Shlendrian has a point. I'm having problems with Raspbian video/web browsers.

Just as a comparison, I have a few different types of small form factor boards. The Tinker Board, even tho it does not have the technical support as does the RPi, and is a bit pricier, outperforms the RPi hands down. I'm watching a 1080p/30fps full screen video streamed from YouTube smooth as silk. It also launches Chromium/widevide v67 with no problem, which allows Netflix and Amazon Prime. Try this video. It is awe-inspiring in 1080p.
https://youtu.be/6v2L2UGZJAM

This is a challenge to the RPi crew. Help me out here. Is there nothing that can be done to make your OS more Debian 9 compatible?
That mostly comes down to the difference in CPU clock speed and RAM difference.

Other boards can handle youtube through brute force, but the pi has always struggled with web browsing in general. In theory, the pi has plenty of power to handle it, but the way existing software is written makes it rather difficult. For example, playing a youtube video isn't as simple as just playing the video. The video is decoded, scaled, youtube playback control overlay is added and the whole thing is rendered into the browser's canvas (that's not exactly how it works, but it's some of the steps involved). That ends up copying a lot of data around and eating a fair bit of RAM, especially if there's format conversion involved. If you don't have the clock cycles, memory and bandwidth to churn through all that, you have to go through and minimise how much data you move around and how many conversions you do.

If you recall, a while back we focused efforts on Epiphany, as it was simplest browser to optimise. A lot of time and money was poured into it, but it was just not a very good browser.

Now the focus is on chromium, but it moves very fast. The optimisations need to be applied to each version, which can take a while. Not a whole lot of people understand the pi, video and chromium well enough to be able to do this and the dev who is is currently working on something else.

It's just not an easy problem to solve.

gkreidl
Posts: 6128
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 10:21 am

ShiftPlusOne has explained it very well. And the process of copying and colour space conversion has never completely worked for full screen on a 1080p screen. It works in a way, but you will have quite a lot of lost frames. It's easy to see when the camera zooms into a complex scene or shifts across a complex background. Some people have claimed that it does work for them (after applying some configuration changes), but they either had a smaller screen size or are simply not able to notice the effect (of lost frames).

I've just fully updated my stretch system (and rebooted because of a new kernel and firmware) and tested youtube in chromium. It works as it has always worked: perfect in normal and cinema mode and with some lost frames in full screen mode. So this is a big fuss about nothing real.

If you open too many tabs in chromium or too many other programs are running and the system has started to swap, it won't work any more, of course.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

SurferTim
Posts: 1769
Joined: Sat Sep 14, 2013 9:27 am
Location: Miramar Beach, Florida

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 12:10 pm

Thanks for your replies. They were helpful.

After a bit of testing, it does appear to be a CPU/GPU problem. Using the "top" app and running Chromium/widevine v67 from Debian, memory appears to be ok, with around 40K free at all times. But the CPU is showing around 300% on a 480p video. The playback has a considerable amount of stuttering.

The Chromium v65 browser included with the upgrade does a bit better, at around 200% CPU on a 480p video, but less available memory, around 32K. The playback has some stuttering, but considerably less than v67.

Using "chrome://gpu" in each browser version, v67 reports hardware acceleration enabled in most categories, but v65 does not. Any way of kicking v65 into hardware acceleration?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 12:30 pm

SurferTim wrote: Using "chrome://gpu" in each browser version, v67 reports hardware acceleration enabled in most categories, but v65 does not. Any way of kicking v65 into hardware acceleration?
I don't think it makes sense. For example, if it uses opengl somewhere, it would use Mesa's software implementation. AFAIK (and I could be wrong), there's nothing that that option would enable that would actually end up using the pi's GPU that doesn't already.

It may make sense with vc4-fkms enabled, but that's still experimental.

Joel's earlier reply may help if you want to try it:
Joel_Mckay wrote: #To enable the chromium GPU acceleration:
# 1. people generally enable the OpenGL module
# 2. and remove "--disable-gpu-compositing'' from /etc/chromium-browser/customizations/00-rpi-var
# Incidentally, I left chromium out of our release given it rendered a webpage too slowly for my taste, ate ram, and the plugin settings page was almost unusable without gpu acceleration enabled.

Schlendrian
Posts: 21
Joined: Wed Oct 03, 2018 5:55 pm

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 1:22 pm

I may also thank you for your replies. It's good to know, that developers are working on it.

I've just done some further testing with my Pi 3B, installed all of the recent updates, and it now works for some odd reason. There is still some noticable slowdown, but it's not as heavy as it was before. But still: on the Pi 3B+ on the other hand, there are heavy slowdowns after installing the updates.

Well, the Pi 3B+ is relatively new, and I guess, these problems will be fixed in near future (Pi 3B had also lots of problems in the beginning - I think, it's absolutely normal, and the 3B works almost flawlessly these days). I would consider using the Pi 3B+ as very experimental. Not really usable for everyday tasks.

SurferTim wrote:
Mon Oct 08, 2018 12:10 pm

The Chromium v65 browser included with the upgrade does a bit better, at around 200% CPU on a 480p video, but less available memory, around 32K. The playback has some stuttering, but considerably less than v67.
How were you able to install Chromium 67? I've tried it a few days ago, but it always tells me, that there's no release candidate for the needed packages on armhf.


Using "chrome://gpu" in each browser version, v67 reports hardware acceleration enabled in most categories, but v65 does not. Any way of kicking v65 into hardware acceleration?
Actually, yes. There are two different ways to bypass this.

1. Install the h264ify extension.

2. Go to chrome://flags, enable "Override software rendering list", "Zero-copy rasterizer" and "GPU rasterization" (you can set this to "Force-enable on all layers").

I don't really know, if the second option still works. I've used this trick about a year ago, when Chromium still had awful screentearing issues. But since they're fixed, I've never touched these options again.
Last edited by Schlendrian on Mon Oct 08, 2018 3:07 pm, edited 1 time in total.

SurferTim
Posts: 1769
Joined: Sat Sep 14, 2013 9:27 am
Location: Miramar Beach, Florida

Re: Newest Kernel and / or Chromium broken

Mon Oct 08, 2018 1:29 pm

Here is a link to the Chromium v67 packages I installed.
https://snapshot.debian.org/package/chr ... :7e:deb9u1
Download the armhf versions. You must rename (mv) to remove the tilde before installing.

Code: Select all

mv chromium_67.0.3396.87-1~deb9u1_armhf.deb chromium_67.0.3396.87-1_deb9u1_armhf.deb
sudo dpkg -i chromium_67.0.3396.87-1_deb9u1_armhf.deb
But beware! This version does not run well on a RPi!

Edit: There may be some dependency issues. My install required this:

Code: Select all

sudo apt-get install libminizip1
sudo apt-get install libre2-3

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 3:59 am

ShiftPlusOne wrote:
Mon Oct 08, 2018 12:30 pm
Joel's earlier reply may help if you want to try it:
Joel_Mckay wrote: #To enable the chromium GPU acceleration:
# 1. people generally enable the OpenGL module
# 2. and remove "--disable-gpu-compositing'' from /etc/chromium-browser/customizations/00-rpi-var
# Incidentally, I left chromium out of our release given it rendered a webpage too slowly for my taste, ate ram, and the plugin settings page was almost unusable without gpu acceleration enabled.
Fortunately the package maintainer as of Oct8, 2018 removed this default parameter,.
Unfortunately, the Rasbian current repo kernel will still hang x11 after a few hours of use (Linux neptune 4.14.70-v7+ #1144 SMP Tue Sep 18 17:34:46 BST 2018 armv7l GNU/Linux ).

Although, the YouTube playback is much smoother than the midori libVLC decoder, the current chromium h264ify plugin will cause an "aw, snap" error... and flash 12 (and also recovered the more recent flash "31.0.0.108") video will cause the bbc videos to also "aw, snap" (the most ambiguous error code I've seen in 16 years ;-) )
Attachments
Screenshot at 2018-10-09 02-26-05.png
Screenshot at 2018-10-09 02-26-05.png (136.59 KiB) Viewed 1725 times
Screenshot at 2018-10-09 02-24-35.png
Screenshot at 2018-10-09 02-24-35.png (51.9 KiB) Viewed 1729 times
Screenshot at 2018-10-09 02-24-16.png
Screenshot at 2018-10-09 02-24-16.png (74.01 KiB) Viewed 1729 times

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 9:37 am

Joel_Mckay wrote: Although, the YouTube playback is much smoother than the midori libVLC decoder, the current chromium h264ify plugin will cause an "aw, snap" error.
Is that ubuntu mate or raspbian with mate installed? Might be running out of RAM?
Joel_Mckay wrote: Unfortunately, the Rasbian current repo kernel will still hang x11 after a few hours of use
Interesting. Initially when I saw this thread I kicked off a playlist on youtube which went on for hours (occasionally switching between full screen and not). Memory kept creeping up, so I thought there might be a leak, but eventually it stabilised and when the tab was closed, went back to normal.

Maybe if I was running a heavier DE like MATE, it would've crashed.

c7borg
Posts: 9
Joined: Thu Apr 26, 2018 7:02 am
Location: United Kingdom

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 10:54 am

Just to add I am also having a similar issue.

Although my setup is harder for anyone else to replicate.

I transcode a UDP live stream to DASH using FFMPEG on a server.

My clients are all raspberry Pis I have about 30 of them dotted around my network.

They all use chromium browser and the javascript shaka player to play the DASH stream.

The Raspberry Pi 3b (jessie) works a treat with 24/7 usage Chrome ver:56.0.2924.84

The Raspberry Pi 3b+ (stretch) works then begins to stutter and drops to 1 frame per 30 seconds this happens after approximately between 20 minutes to 40 minutes my current build uses Chrome Ver:65.0.3325.181

I have also tried stretch on the 3b and the same happens

I have also tried Chrome version 56.0.2924.84 on the 3b+ using stretch the same issues

This would point to stretch itself - I'm currently using the latest release but am at a bit of a loss where to look next but open to ideas

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 10:57 am

c7borg wrote:
Wed Oct 10, 2018 10:54 am
Just to add I am also having a similar issue.

Although my setup is harder for anyone else to replicate.

I transcode a UDP live stream to DASH using FFMPEG on a server.

My clients are all raspberry Pis I have about 30 of them dotted around my network.

They all use chromium browser and the javascript shaka player to play the DASH stream.

The Raspberry Pi 3b (jessie) works a treat with 24/7 usage Chrome ver:56.0.2924.84

The Raspberry Pi 3b+ (stretch) works then begins to stutter and drops to 1 frame per 30 seconds this happens after approximately between 20 minutes to 40 minutes my current build uses Chrome Ver:65.0.3325.181

I have also tried stretch on the 3b and the same happens

I have also tried Chrome version 56.0.2924.84 on the 3b+ using stretch the same issues

This would point to stretch itself - I'm currently using the latest release but am at a bit of a loss where to look next but open to ideas
You could try rolling back the kernel and firmware individually to check if it's either of those. Then bisect to find the exact version where the problem was introduced.

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 1:15 pm

ShiftPlusOne wrote:
Wed Oct 10, 2018 9:37 am
Interesting. Initially when I saw this thread I kicked off a playlist on youtube which went on for hours (occasionally switching between full screen and not). Memory kept creeping up, so I thought there might be a leak, but eventually it stabilised and when the tab was closed, went back to normal.
Maybe if I was running a heavier DE like MATE, it would've crashed.
That was probably without the gl module active using marco... ;-)
The vc4 devs were kind enough to explain the failure mode back in Jul, and provided an effective (albiet imperfect) workaround.
In one of the scenarios: Given enough time, the gpu and cpu dynamic shared memory Allocator will fragment the layout and eventually throw an error that freezes x11 (kernel will continue to operate).

Indeed, we spent some time stripping down the MATE desktop on a standard Raspbian Stretch Lite base, and you can see in the above photo chromium is running without the h264ify plugin at about 510MB of 875MB (desktop itself reports as 260MB using "sudo ps_mem" python utility).
I have attached some other benchmarks with the custom RT kernel, but keep in mind the held VLC/libVLC version used in midori is the custom gl patched version that is minimally functional. The photos shown will not be repeatable with a regular Raspbian repo packaged install.
c7borg wrote:
Wed Oct 10, 2018 10:54 am
This would point to stretch itself - I'm currently using the latest release but am at a bit of a loss where to look next but open to ideas
Our club is primarily focused on version-locked RTlinux based hardware, but the patched kernel has tested at over 10 days up-time with the gpu compositor... you can click my user url on the right, if you would like to try to use it to help diagnose your issue (note the documented reload-once-browser plugin bug is a manifestation of the aforementioned workaround).
ShiftPlusOne wrote:
Wed Oct 10, 2018 10:57 am
You could try rolling back the kernel and firmware individually to check if it's either of those. Then bisect to find the exact version where the problem was introduced.
Unfortunately, this will unlikely work if it is indeed the same vc4 bug I encountered and reported in Jul .

Cheers,
~J~
Attachments
2.jpeg
2.jpeg (169.63 KiB) Viewed 1652 times
1.jpeg
1.jpeg (154.72 KiB) Viewed 1652 times

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: Newest Kernel and / or Chromium broken

Wed Oct 10, 2018 1:58 pm

There was a very recent fix for a CMA (IIRC) fragmentation issue that eventually caused things, I think in Kodi, but probably elsewhere as well, to grind to a halt. Could this be related?

I believe it was an upstream kernel change that exposed the issue of small memory allocations bunging things up. Fixed by adding a pool allocator and reverting the kernel change. Now that I think about it a bit more, maybe not related, but perhaps worth consideration.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 1:48 am

jamesh wrote:
Wed Oct 10, 2018 1:58 pm
There was a very recent fix for a CMA (IIRC) fragmentation issue that eventually caused things, I think in Kodi, but probably elsewhere as well, to grind to a halt. Could this be related?

I believe it was an upstream kernel change that exposed the issue of small memory allocations bunging things up. Fixed by adding a pool allocator and reverting the kernel change. Now that I think about it a bit more, maybe not related, but perhaps worth consideration.
Not sure, but I have determined the current Oct8,2018 Rasbian repo firmware update and pulse now also causes HDMI audio output to cease functioning after powering off the HDMI tv for a moment (Jul firmware and same kernel did not do this). I am currently looking into this new issue to try to diagnose the specific origin is indeed either the new firmware or the pulse package.

Personally, I would suggest reviewing the regression unit-test framework around the Pi firmware project... ;-) Might I suggest this footage for the beta tests... given it seems rather anecdotal for myself :-)
https://www.youtube.com/watch?v=c7gxStnBfbg

Cheers,
~J~
--
tail -n30 /var/log/syslog
Oct 11 02:46:08 neptune pulseaudio[1430]: [alsa-sink-MAI PCM vc4-hdmi-hifi-0] alsa-sink.c: Error opening PCM device iec958:0: Device or resource busy
Oct 11 02:46:08 neptune pulseaudio[1430]: [pulseaudio] sink-input.c: Failed to create sink input: sink is suspended.

Bug report bread-crumb: https://github.com/raspberrypi/linux/issues/2710

c7borg
Posts: 9
Joined: Thu Apr 26, 2018 7:02 am
Location: United Kingdom

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 8:31 am

Joel_Mckay wrote:
Wed Oct 10, 2018 1:15 pm
c7borg wrote:
Wed Oct 10, 2018 10:54 am
This would point to stretch itself - I'm currently using the latest release but am at a bit of a loss where to look next but open to ideas
Our club is primarily focused on version-locked RTlinux based hardware, but the patched kernel has tested at over 10 days up-time with the gpu compositor... you can click my user url on the right, if you would like to try to use it to help diagnose your issue (note the documented reload-once-browser plugin bug is a manifestation of the aforementioned workaround).

Cheers,
~J~
Thanks Joel,

I would really like to give this a try, I've had a look at your site, could you point me to the patched kernel, I'm assuming that once you point me to the file I just need to do a 'sudo rpi-update' ?

Is that the only change?

Many thanks
Andy.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 8:33 am

Joel_Mckay wrote:
Thu Oct 11, 2018 1:48 am
jamesh wrote:
Wed Oct 10, 2018 1:58 pm
There was a very recent fix for a CMA (IIRC) fragmentation issue that eventually caused things, I think in Kodi, but probably elsewhere as well, to grind to a halt. Could this be related?

I believe it was an upstream kernel change that exposed the issue of small memory allocations bunging things up. Fixed by adding a pool allocator and reverting the kernel change. Now that I think about it a bit more, maybe not related, but perhaps worth consideration.
Not sure, but I have determined the current Oct8,2018 Rasbian repo firmware update and pulse now also causes HDMI audio output to cease functioning after powering off the HDMI tv for a moment (Jul firmware and same kernel did not do this). I am currently looking into this new issue to try to diagnose the specific origin is indeed either the new firmware or the pulse package.

Personally, I would suggest reviewing the regression unit-test framework around the Pi firmware project... ;-) Might I suggest this footage for the beta tests... given it seems rather anecdotal for myself :-)
https://www.youtube.com/watch?v=c7gxStnBfbg

Cheers,
~J~
--
tail -n30 /var/log/syslog
Oct 11 02:46:08 neptune pulseaudio[1430]: [alsa-sink-MAI PCM vc4-hdmi-hifi-0] alsa-sink.c: Error opening PCM device iec958:0: Device or resource busy
Oct 11 02:46:08 neptune pulseaudio[1430]: [pulseaudio] sink-input.c: Failed to create sink input: sink is suspended.

Bug report bread-crumb: https://github.com/raspberrypi/linux/issues/2710
Good luck with trying to implement a test framework that tests everything for a linux machine that does....everything.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 10:26 am

jamesh wrote:
Thu Oct 11, 2018 8:33 am
Good luck with trying to implement a test framework that tests everything for a linux machine that does....everything.
No offense was intended, and hopefully my input was not considered derogatory :-) (my communication skills often don't articulate my intentions very well...)
I am surprised an organization that produces millions of units does not have production line automated diagnostics access already available to the devs. On another platform, historically we would test using "expect" to script "xdotool" for macro input, use "xwd -out filename.xwd" to extract a screen buffer snapshot (some used xvfb+vglrun for headless), and compare it against a Beta tester's cropped reference snapshot with "visgrep" (part of "xautomation"). Even just opening a file with a program and "grep -c" the syslog before and after to check for error count changes (and cpu average load for a minute) is very easy.

The idea was to verify core functionality was repeatable during each incremental update to the OS and program builds (i.e. be as unsurprising as possible for users, and auto-detect bad kernel patch levels before deploying it on the cluster). Detecting changes that manifest when a dev unintentionally changes something undocumented is complex unless a working baseline is used. There are also dozens of GUI testing frameworks like https://ldtp.freedesktop.org/wiki/ which can be coded more cleanly, but I prefer scripts appending to a log because it allows parsing VM/VNC windows with just about any OS running.

For example:
a. killall instances of an application you want to run
b. grep log for panic/error/fail message count
c. open webpage full-screen using default browser application
d. wait 20 seconds to download page
e. capture window/desktop contents as image
f. determine degree of match (gui is as expected?)
g. close program using macro
h. wait 10 seconds
i. check process list for lingering application (also, remote shell may freeze if OS crashed)
j. kill -9 pid of abnormal application
k. grep log again for new panic/error/fail message count, and determine change in count
l. repeat test 3 times for Rasbian base applications everyone will use


@c7borg
The kernel we use is setup to use a ramdrive overlayfs, and is not really appropriate for the regular Raspbian distro. The regular mainline OS will get much better as the rough parts get polished smooth. I may abandon the local thread given the topic has begun to drift a bit... which is unfair to the original participants.

Cheers,
~J~

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 11:07 am

Joel_Mckay wrote:
Thu Oct 11, 2018 10:26 am
jamesh wrote:
Thu Oct 11, 2018 8:33 am
Good luck with trying to implement a test framework that tests everything for a linux machine that does....everything.
No offense was intended, and hopefully my input was not considered derogatory :-) (my communication skills often don't articulate my intentions very well...)
I am surprised an organization that produces millions of units does not have production line automated diagnostics access already available to the devs. On another platform, historically we would test using "expect" to script "xdotool" for macro input, use "xwd -out filename.xwd" to extract a screen buffer snapshot (some used xvfb+vglrun for headless), and compare it against a Beta tester's cropped reference snapshot with "visgrep" (part of "xautomation"). Even just opening a file with a program and "grep -c" the syslog before and after to check for error count changes (and cpu average load for a minute) is very easy.

The idea was to verify core functionality was repeatable during each incremental update to the OS and program builds (i.e. be as unsurprising as possible for users, and auto-detect bad kernel patch levels before deploying it on the cluster). Detecting changes that manifest when a dev unintentionally changes something undocumented is complex unless a working baseline is used. There are also dozens of GUI testing frameworks like https://ldtp.freedesktop.org/wiki/ which can be coded more cleanly, but I prefer scripts appending to a log because it allows parsing VM/VNC windows with just about any OS running.

For example:
a. killall instances of an application you want to run
b. grep log for panic/error/fail message count
c. open webpage full-screen using default browser application
d. wait 20 seconds to download page
e. capture window/desktop contents as image
f. determine degree of match (gui is as expected?)
g. close program using macro
h. wait 10 seconds
i. check process list for lingering application (also, remote shell may freeze if OS crashed)
j. kill -9 pid of abnormal application
k. grep log again for new panic/error/fail message count, and determine change in count
l. repeat test 3 times for Rasbian base applications everyone will use


@c7borg
The kernel we use is setup to use a ramdrive overlayfs, and is not really appropriate for the regular Raspbian distro. The regular mainline OS will get much better as the rough parts get polished smooth. I may abandon the local thread given the topic has begun to drift a bit... which is unfair to the original participants.

Cheers,
~J~
We do have production line testing, but none of the results from that come back to us - they simply test the HW is working as expected. I don't think any firmware is uploaded or run - the production speed required prohibits that. Or do you actually mean production line testing, because you go on to talk about what I woudl regard as something else.

I do believe personally we need to improve our in house testing of software changes but we have only a few engineers to do a lot of work. We have I think 23 or 24 engineers, spread over HW and software and we are all busy on all sorts of projects. But testing is on the list of things to improve.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 11:13 am

jamesh wrote:
Thu Oct 11, 2018 11:07 am
I do believe personally we need to improve our in house testing of software changes.
+1. Right now it checks that the firmware boots on all pi models, but should really be extended to test whole images and features thereof.

I made a start on a system that used pi zero in gadget mode to act as a USB stick with a test image on it. GPIO was connected to the RUN pins. I had it booting the latest image and switching over to composite mode to test network functionality as well, but had to switch to working on something else.

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 11:44 am

ShiftPlusOne wrote:
Thu Oct 11, 2018 11:13 am
I made a start on a system that used pi zero in gadget mode to act as a USB stick with a test image on it. GPIO was connected to the RUN pins. I had it booting the latest image and switching over to composite mode to test network functionality as well, but had to switch to working on something else.
I looked around for HDMI capture cards compatible with a Debian/Ubuntu PC host to run the auditing terminal script host.
https://inogeni.com/product/4k-to-usb-3 ... ifications
These are typically used for people recording videogames, but should allow you to setup an accessible automated test rig.

@jamesh
I was getting a little too verbose with pseudo-code, but we used "expect" as the tcl shell can automate a remote "ssh" login and terminal interactions.
if you have never encountered the tool, I can see that it would sound a bit like a non sequitur. ;-)

Cheers,
~J~

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 11:48 am

Joel_Mckay wrote:
Thu Oct 11, 2018 11:44 am
ShiftPlusOne wrote:
Thu Oct 11, 2018 11:13 am
I made a start on a system that used pi zero in gadget mode to act as a USB stick with a test image on it. GPIO was connected to the RUN pins. I had it booting the latest image and switching over to composite mode to test network functionality as well, but had to switch to working on something else.
I looked around for HDMI capture cards compatible with a Debian/Ubuntu PC host to run the auditing terminal script host.
https://inogeni.com/product/4k-to-usb-3 ... ifications
These are typically used for people recording videogames, but should allow you to setup an accessible automated test rig.
We'd just use fbgrab or scrot, and send that over for checking. We also have ways of capturing data through the DSI slot rather than HDMI, if necessary.

None of it is terribly difficult, but somebody needs to set aside a chunk of time to actually do the work and then maintain it.

User avatar
Joel_Mckay
Posts: 289
Joined: Mon Nov 12, 2012 10:22 pm
Contact: Website

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 12:16 pm

ShiftPlusOne wrote:
Thu Oct 11, 2018 11:48 am
We'd just use fbgrab or scrot, and send that over for checking. We also have ways of capturing data through the DSI slot rather than HDMI, if necessary.

None of it is terribly difficult, but somebody needs to set aside a chunk of time to actually do the work and then maintain it.
I assumed verification of HDMI Audio, Video, and gl rendering using a standard off-the-shelf driver-less device would save resources for you guys. Note, screen fb grabs worked for our team as it was purely a quick software testing sub-project, but controlling variables by testing as the devices will be used... will help minimize your teams surprises (and trust it will ultimately save your team time).
;-)

Best of luck,
~J~

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6074
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Newest Kernel and / or Chromium broken

Thu Oct 11, 2018 12:31 pm

Joel_Mckay wrote: I assumed verification of HDMI Audio, Video, and gl rendering using a standard off-the-shelf driver-less device would save resources for you guys.
Good point. I was thinking just of the things I'm responsible for but of course it would make sense to test HDMI.

Bf546
Posts: 1
Joined: Sun Mar 17, 2019 2:01 am

Re: Newest Kernel and / or Chromium broken

Sun Mar 17, 2019 2:34 am

Tell me if you think this is related. I've been trying to figure this out for too long.

We use hundreds of pis in a production environment. They are built with the most simple configuration, raspbian, xorg, openbox, fbpanel, and chromium. The employees use chromium to run our internal apps.

The image I built approximately a year ago, with kernel 4.9, seems to work quite well. Somewhere down the road, Raspbian decided to update the kernel within its Stretch repository to 4.14... a HUGE no no seeing how Debian's stable branch is only to receive security updates. Yeah I get it, they wanted to make it so people could boot the 3B+.

Anyway, the behavior change: after using apt to download and install security updates, we've noticed a lot of random network pausing in chromium, up to 2 minutes at a time. It's not the entire network that pauses though. You can ping the pi and get uninterrupted replies. Ran tcpdump and studied multiple captures. When the network timeouts happen, the pi is actually trying to establish a connection with one of our Ajax servers. The funny thing however, the pi is waiting on the SYN ACK response from one of the servers. Although that might seem like the server is misconfigured, running our app on chromium on a Windows machine, a Linux machine using an x86 architecture, or the old pi image I made a year ago, the problem just will not occur.

For a while I thought it was a chromium problem. To rule it out we installed the kweb browser. Ran our internal app in the kweb browser and it resulted in the same behavior.

I don't believe it's a chromium problem, I believe it's very possibly a kernel problem. I also tried changing sources.list to buster. Did a dist-upgrade which upgraded the distribution to buster. Kernel version is also upgraded to 4.19. same results, problem hasn't been fixed yet.

For now we're sticking to the original image. It's not completely necessary to install updates but it would be nice.

Return to “Troubleshooting”