Extra 16M memory on Pi2/Pi3


10 posts
by dom » Wed Apr 12, 2017 4:51 pm
The Pi has a 30-bit (1024MB) address space (the top two bits are used for caching aliases).
A small amount of one of the aliases is used for gpu peripheral access.

The arm accesses gpu memory space through a coarse MMU that allows 64 blocks of 16M to be mapped in.
On Pi0/Pi1 we use 32 of these blocks to access 512M of sdram and one block for gpu peripherals.
On Pi2/Pi3 we use 63 of these blocks to access 1008M of sdram and one block for gpu peripherals.

Because of this mapping on Pi2/Pi3 the top 16M of sdram cannot be seen by the arm and is currently not used.

It makes sense for the gpu to use this top 16M of sdram, but there are complications.
Some parts of gpu memory are accessed from the arm, e.g. the touchscreen and led drivers use a shared block of memory.
Debugging tools like vcdbg read gpu memory to extract gpu logs and dumps of the memory heaps.

I believe I have worked around these issues (using arm allocated buffers for communication and using dma to access gpu memory from vcdbg).

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:
Code: Select all
total_mem=1024

The extra 16M will be enabled. If gpu_mem is not specified we will increase gpu memory from 64M to 76M and arm memory from 944M to 948M
as the extra gpu memory improves Chromium's hardware video decode performance.

You are free to explicitly set it (e.g. gpu_mem=64) if you prefer the extra memory to purely be on the arm.

You can test the memory available with:
Code: Select all
vcgencmd get_mem arm
vcgencmd get_mem gpu


Notes:
This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty.
I have seen reports of crashes occurring with "total_mem=1024" from kodi users. Those users have reported "total_mem=1023" is stable.
I haven't been able to reproduce the crash. Be aware that this setting is experimental and could cause instability so don't enable on a critical system.
Please start with "total_mem=1024". Instructions for how to reproduce a crash with this would be very useful.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5076
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by timrowledge » Wed Apr 12, 2017 5:33 pm
Interesting idea, and I have a very faintly possible explanation why using 1024 instead of 1023 might cause an issue. Do remember that "very faintly possible" part.

This derives from experience in the Mac version of Squeak Smalltalk where graphics are generated and bitblt'd (remember, BitBLT was invented by Dan Ingalls for Smalltalk) to a bitmap buffer. If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have. Boom, as Ivanova likes to put it.

Maybe, just maybe, Kodi occasionally does something similar.
Making Smalltalk on ARM since 1986; making your Scratch better since 2012
Posts: 1031
Joined: Mon Oct 29, 2012 8:12 pm
Location: Vancouver Island
by dom » Wed Apr 12, 2017 5:43 pm
timrowledge wrote:If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have.


It could be something like that. In general if user code touches a word beyond the last page you allocated I'd expect linux to give you a page fault and kill your process (with or without the 1024M).
But there may be things in the kernel where permissions are more relaxed (e.g. using DMA hardware) where reading beyond a 4K page is tolerated but beyond 1024M may be more fatal.

Hopefully someone will test this and find an application that fails reliably. That would be useful in tracking it down.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5076
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by cjan » Sat Apr 15, 2017 1:32 am
in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.
Posts: 409
Joined: Sun May 06, 2012 12:00 am
by tvjon » Sat Apr 15, 2017 12:07 pm
"This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty."

A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

I've been experimenting with vlc recently, in particular using its mmal video output, so popped my pi3's µSD card into the 0.

Attached is the result of trying to stream from an ip camera.

I then remembered I'd changed config.txt to use your new extra ram setting, so I changed it to fixed gpu value. 32 megs always results in a seg' fault when vlc is started.

I tried other values up to 256 meg's, all of which allowed vlc to load fine, but always display a black screen where the video should be.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.

vlc does allocate a lot of buffers, so I'll rebuild it with fewer to see if that lets it show the screen properly.


pi0.jpg
pi0.jpg (60.31 KiB) Viewed 9079 times
Posts: 520
Joined: Mon Jan 07, 2013 9:11 am
by dom » Sun Apr 16, 2017 11:34 am
cjan wrote:in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.

That won't make any difference for video playback in chromium.

For kodi, then sure, more gpu memory is useful (256MB is recommended for Pi2/Pi3).
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5076
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by dom » Sun Apr 16, 2017 11:37 am
tvjon wrote:A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.

Don't set total_mem=1024 on a Pi0/Pi1 - it makes no sense and it will be clamped to 512 anyway.
If you have trouble running vlc then create a new thread - it has nothing to do with this one.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5076
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by zipplet » Sat Jun 24, 2017 12:53 pm
I have been using this setting with retropie + the latest updates to Raspbian without issue so far at 1024 on a Pi 3B - I'll report back if I manage to get it to crash. Retropie is a good stress test with some of the emulators.

One other way to play with this would be to use that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel - then the theories about touching bad memory addresses could be proven/etc.
Posts: 13
Joined: Sun Dec 11, 2016 4:05 pm
Location: Tokyo, Japan
by B.Goode » Sat Jun 24, 2017 1:16 pm
that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel


Ultibo?

https://ultibo.org
Posts: 3744
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK
by runboy93 » Sat Jun 24, 2017 7:25 pm
768M arm, 256M gpu

config.txt should have "total_mem=1024" by default, or better informed, it came as little surprise for me too while ago.

16M may sound very little, but with RPi every bit is always a bonus.

That crash with kodi was maybe just very rare case it happened, atleast haven't seen myself nor on internet talking about crash on Raspbian because of this tweak.
RPi 3 tweaks by runboy93
https://pastebin.com/raw/TBzXDUp0
Posts: 171
Joined: Tue Feb 28, 2017 1:17 pm
Location: Finland