6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7513
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: VC4 and Mesa Drivers Update Plan

Wed Sep 26, 2018 12:08 pm

Neex wrote:
Wed Sep 26, 2018 9:58 am
Just putting my hand up saying that we've invested heavily in RasPi for our product, and Kivy as the provider for our UI. As we near product launch, and discover that our UI is failing because of a memory leak, we are extremely concerned. Can anyone provide a status, schedule and workaround? Many thanks.
One could say that basing anything critical on something that is very openly described as experimental is a touch foolhardy.

Workaround - build mesa and libdrm for yourselves. https://github.com/anholt/mesa/wiki/Bui ... sa-for-VC4

Schedule - there is a backport of mesa 18.1.6 for Raspbian Stretch (https://packages.debian.org/stretch-bac ... begl-mesa0).
If someone fancied either
a) testing 18.1.6
b) providing a SIMPLE test case that can be tested by others
then it may be possible to update Raspbian to that (it all depends on the number of dependencies that also need updating).
Stability is the name of the game. It's nigh on impossible to test everything that relies on Mesa, hence the hand-off approach to updating it randomly.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Gavinmc42
Posts: 4020
Joined: Wed Aug 28, 2013 3:31 am

Re: VC4 and Mesa Drivers Update Plan

Wed Dec 19, 2018 5:40 am

Back on June 27th I complained about the Gentoo64 Mesa gear demo being jerky when the window was made bigger.
Not anymore, full screen is runs smooth and only drops down to 55fps.
Terrain demo also works full screen and around 50fps :D
Time to test some OpenGL games?

So is Kivy still a problem?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

shivansps
Posts: 35
Joined: Thu Jan 26, 2012 5:11 pm

Re: VC4 and Mesa Drivers Update Plan

Fri Jan 04, 2019 12:40 pm

Gavinmc42 wrote:
Wed Dec 19, 2018 5:40 am
Back on June 27th I complained about the Gentoo64 Mesa gear demo being jerky when the window was made bigger.
Not anymore, full screen is runs smooth and only drops down to 55fps.
Terrain demo also works full screen and around 50fps :D
Time to test some OpenGL games?

So is Kivy still a problem?
Well i got Freespace 2 working viewtopic.php?f=78&t=230316

thats kinda the most complex and 3D intensive game that had been run on a PI i think.

I want to make some guide son installing some freeware games that run on the freespace 2 open engine but i need to at least to get software S3TC texture compresion working for that. What i was not able to do even trought the correct mesa lib is installed.

User avatar
Gavinmc42
Posts: 4020
Joined: Wed Aug 28, 2013 3:31 am

Re: VC4 and Mesa Drivers Update Plan

Sat Jan 05, 2019 10:56 am

thats kinda the most complex and 3D intensive game that had been run on a PI i think.
You could be right in that, not sure about the texture stuff in the Quake port.
I need to have look now that I understand textures a bit more

Tim Skillmam (tipam) has been doing great stuff with OpenGLES, not quite a game engine yet.
He has been doing some baking on his Star Citizen ship.
viewtopic.php?f=68&t=214695

I am now wondering if ETC1 can be used to get more compression?

Kivy uses S3TC?
https://www.khronos.org/registry/OpenGL ... n_s3tc.txt

From the Videocore manual
Each TMU is ‘virtualized’ and therefore can theoretically sample from an unlimited number of textures in any
given shader
Use the Arm core to convert S3TC to ETC1 on startup of the game.
Because ETC1 is compressed it might use less memory than pure bitmaps.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

FrankGould
Posts: 27
Joined: Mon Mar 12, 2018 2:00 pm

Re: VC4 and Mesa Drivers Update Plan

Mon Feb 25, 2019 4:33 pm

I spent the first half of 2018 migrating my Android 4.4.2 SlideShow app, written in Python and using Kivy UI module, that I first reported a problem with the Kivy screenmanager memory leaks, originally posted in this thread. This leak prevents my app from running more than a minute before crashing and locking up the system .

After several tests across multiple linux distributions, like Ubuntu Mate 16.04 and 18.04, Gentoo, and Arch, we determined there are graphics driver problem on all of them. My buddy Kuzeyron and I have tried all possible combinations of graphics driver (legacy, KMS, and fake KMS) with these distributions. Kuzeyron has even attempted to compile the open source versions to update the drivers with newer releases but was unable to get the drivers to install correctly.

At this point, I have fallen back to my pi3d driven SlideShow app so that it can manage the GPU memory and reboot Raspbian after months running continuously. Going forward, we would like to find someone who can help us get past this graphics driver memory leak in Raspbian so that the Kivy screenmanager works without crashing the Raspberry Pi.

Is there anyone who can help us with this anomaly? If so, you can usually find one of us on the Kivy channel on Discord. :cry:

cjan
Posts: 742
Joined: Sun May 06, 2012 12:00 am

Re: VC4 and Mesa Drivers Update Plan

Mon Feb 25, 2019 11:44 pm

did you try kernel-4.19 & mesa-18.3.4 ?

FrankGould
Posts: 27
Joined: Mon Mar 12, 2018 2:00 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Feb 26, 2019 1:17 pm

@cjan, we have tried everything we possibly can but are not aware how to try "kernel-4.19 & mesa-18.3.4." We're looking for ways to do that. Can you point us in the right direction?

Like I said, we tried compiling the drivers but cannot get them to load. Any tutorials or instructions how to do that would be helpful.

cjan
Posts: 742
Joined: Sun May 06, 2012 12:00 am

Re: VC4 and Mesa Drivers Update Plan

Wed Feb 27, 2019 12:05 am

FrankGould wrote:
Tue Feb 26, 2019 1:17 pm
@cjan, we have tried everything we possibly can but are not aware how to try "kernel-4.19 & mesa-18.3.4." We're looking for ways to do that. Can you point us in the right direction?

Like I said, we tried compiling the drivers but cannot get them to load. Any tutorials or instructions how to do that would be helpful.
but you tried Arch above post? and Alarm have kernel-4.19.25 & mesa-18.3.4.
they had some VC4 commit both.

cjan
Posts: 742
Joined: Sun May 06, 2012 12:00 am

Re: VC4 and Mesa Drivers Update Plan

Wed Feb 27, 2019 1:48 am

and more commit mesa-19.0 next week.
https://cgit.freedesktop.org/mesa/mesa/ ... grep&q=v3d

FrankGould
Posts: 27
Joined: Mon Mar 12, 2018 2:00 pm

Re: VC4 and Mesa Drivers Update Plan

Fri Mar 01, 2019 3:11 pm

Yes, we tried ALARM and the results were the same with memory leaks, roughly 10K per hour over about 4 hours. The kernel and mesa versions you reported are accurate. I've heard things might change next week and would like to know if there is anything we can do to prepare for the release, like make test plans?

Also, is there anyone in dev who can comment on any of this release or anything? Would be nice to get some ideas what's coming sooner or later. :ugeek:

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7513
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: VC4 and Mesa Drivers Update Plan

Fri Mar 01, 2019 4:58 pm

The kernel packaged in Raspbian will be bumped from 4.14 to 4.19 once the final few niggles over Wifi have been resolved. I'd hope that would be within the next month, but it's not set in stone.
If you use rpi-update at present you will get a 4.19 kernel. Feel free to test it now.

Raspbian is still based on Debian Stretch, so mesa won't be updated unless upstream update it.
Debian Buster has just passed a soft freeze, with hard freeze due in a couple of weeks. Final release is expected in June or July. Exactly when we will have a Raspbian Buster image ready for release is currently unknown.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

FAGould
Posts: 21
Joined: Thu Aug 08, 2019 1:10 pm

Re: VC4 and Mesa Drivers Update Plan

Thu Aug 08, 2019 1:40 pm

It appears the Buster graphics drivers still have a memory leak. I ran screenmanagertest.py (https://gist.github.com/frankgould/2464 ... f9e849524b) overnight with 1366x768 res pictures on full screen and started at 5PM from Used at 204K up to 212K at 8:21AM. The app ran fine with no logged errors or warnings.

I will setup a dedicated RPi4B/4GB to let it run for days. I will report back once a few days have passed to share what the results are.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24117
Joined: Sat Jul 30, 2011 7:41 pm

Re: VC4 and Mesa Drivers Update Plan

Thu Aug 08, 2019 2:19 pm

So, those numbers are a bit odd. You ran for 15h21 minutes (5pm->8.21am), which is 921 minutes and leaked 8k. Your code seem to display something every 5 seconds, and 921 minutes is 11052 of those loads, so that less than one byte lost per load. So doesn't appear to be related to the number of times the image load is done. So it's possibly a single buffer allocated at some point. But when? Does the memory appeared leaked straight away, or is is gradually increasing?

Why do you think it's the graphics stack leaking? Rather than Python, or something else on the system?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
Gavinmc42
Posts: 4020
Joined: Wed Aug 28, 2013 3:31 am

Re: VC4 and Mesa Drivers Update Plan

Fri Aug 09, 2019 2:17 am

Why do you think it's the graphics stack leaking? Rather than Python, or something else on the system?
I gave up on Python with PyGame about 4-5 years ago.
And ditto for Raspbian Linux 4 years ago.
Just too hard to find out why the memory was filling up and crashing.

I only have one old model A, not net connected that runs that Raspbian/Python combo.
Model B's with same code on a network are flaky.
Too many things to debug for me, I gave up.

I do know since Gentoo64 came out with newer mesa things got a bit better.
And Buster is looking pretty good, but I'm still not sure I trust Python.

Some solutions I used are PiCore, shell script and a bit of Micropython, that is rock solid.

None of which helps you at all.
Except I know how you feel.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

FAGould
Posts: 21
Joined: Thu Aug 08, 2019 1:10 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 12:07 pm

I will add my experience to this thread because it appears the graphics and displays on Raspberry Pis are not high priority. As for network stability, I've found that Arch Linux ARM is incredibly reliable in my tests. In the link below, you can see three RPi3A+ models running a rover, remote control, and Wi-Fi access point with GPS for RTC and location logging. The only graphics are on the remote control that uses kivy for the GUI. The access point has been running for two months now and has been up when our Apple Wi-Fi router loses connections during the day. We have to constantly restore our mobile devices to the home network when they time out on the Apple LAN.

https://sites.google.com/view/goddard-r ... ype-1-2019

So, I am hopeful that one day the graphics and displays (https://github.com/raspberrypi/firmware ... -520525448) will be fixed and/or work as they had. Raspberry Pi is a great concept and the foundation has delivered products that satisfied my basic requirements but still need tweaking to be fully functional.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24117
Joined: Sat Jul 30, 2011 7:41 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 12:18 pm

Not sure where you get the idea they are not high priority - there's not much higher. We have 2 1/2 internal engineers working on display stuff right now. It is holiday time however, I'm off for the next couple of weeks.

We also are contracting in two companies who are working on the Mesa/VC4/DRM/KMS stuff as they are domain experts - if you look at the checkins you will see that they have been fairly busy.

So on the whole, more engineering is being put in to display stuff than any other areas.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

FAGould
Posts: 21
Joined: Thu Aug 08, 2019 1:10 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 12:25 pm

Thanks jamesh. Below is why I said not high priority. Your answer helps explain going forward and that's great. I'm just trying to migrate my app and will have to be patient for updates. Thanks for your contributions!

"It is something we are aware of but it's not top of the list of issues. Basically the HDMI hardware is different on Pi4 and 4kp60 scrambling adds another layer of difficulty to achieving this."

https://github.com/raspberrypi/firmware ... -520525448

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24117
Joined: Sat Jul 30, 2011 7:41 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 12:39 pm

FAGould wrote:
Tue Aug 13, 2019 12:25 pm
Thanks jamesh. Below is why I said not high priority. Your answer helps explain going forward and that's great. I'm just trying to migrate my app and will have to be patient for updates. Thanks for your contributions!

"It is something we are aware of but it's not top of the list of issues. Basically the HDMI hardware is different on Pi4 and 4kp60 scrambling adds another layer of difficulty to achieving this."

https://github.com/raspberrypi/firmware ... -520525448
TBH, in the list of display things, that is low priority. We have much bigger fish to fry (sorting out why some monitors EDIDs are not fully recognised is a priority). Although I think there might be some news on the power off thing in the next few days.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7513
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 12:43 pm

FAGould wrote:
Tue Aug 13, 2019 12:25 pm
Thanks jamesh. Below is why I said not high priority. Your answer helps explain going forward and that's great. I'm just trying to migrate my app and will have to be patient for updates. Thanks for your contributions!

"It is something we are aware of but it's not top of the list of issues. Basically the HDMI hardware is different on Pi4 and 4kp60 scrambling adds another layer of difficulty to achieving this."

https://github.com/raspberrypi/firmware ... -520525448
That comment is in relation to the display not going tj standby when the screen saver kicks in, which isnt really a critical feature compared to eg display tearing or other more visible artifacts.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

FAGould
Posts: 21
Joined: Thu Aug 08, 2019 1:10 pm

Re: VC4 and Mesa Drivers Update Plan

Tue Aug 13, 2019 1:05 pm

That comment is in relation to the display not going tj standby when the screen saver kicks in, which isnt really a critical feature compared to eg display tearing or other more visible artifacts.
@6by9 My comment was about vcgencmd turning off/on the monitor, like RPi3B+ does. The github #3050 is about the display turning off the screen when screen save kicks in and not blanking the screen with the backlight still on. Two different issues about HDMI displays.

Return to “Raspbian”