dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Extra 16M memory on Pi2/Pi3

Wed Apr 12, 2017 4:51 pm

The Pi has a 30-bit (1024MB) address space (the top two bits are used for caching aliases).
A small amount of one of the aliases is used for gpu peripheral access.

The arm accesses gpu memory space through a coarse MMU that allows 64 blocks of 16M to be mapped in.
On Pi0/Pi1 we use 32 of these blocks to access 512M of sdram and one block for gpu peripherals.
On Pi2/Pi3 we use 63 of these blocks to access 1008M of sdram and one block for gpu peripherals.

Because of this mapping on Pi2/Pi3 the top 16M of sdram cannot be seen by the arm and is currently not used.

It makes sense for the gpu to use this top 16M of sdram, but there are complications.
Some parts of gpu memory are accessed from the arm, e.g. the touchscreen and led drivers use a shared block of memory.
Debugging tools like vcdbg read gpu memory to extract gpu logs and dumps of the memory heaps.

I believe I have worked around these issues (using arm allocated buffers for communication and using dma to access gpu memory from vcdbg).

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024
The extra 16M will be enabled. If gpu_mem is not specified we will increase gpu memory from 64M to 76M and arm memory from 944M to 948M
as the extra gpu memory improves Chromium's hardware video decode performance.

You are free to explicitly set it (e.g. gpu_mem=64) if you prefer the extra memory to purely be on the arm.

You can test the memory available with:

Code: Select all

vcgencmd get_mem arm
vcgencmd get_mem gpu
Notes:
This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty.
I have seen reports of crashes occurring with "total_mem=1024" from kodi users. Those users have reported "total_mem=1023" is stable.
I haven't been able to reproduce the crash. Be aware that this setting is experimental and could cause instability so don't enable on a critical system.
Please start with "total_mem=1024". Instructions for how to reproduce a crash with this would be very useful.

timrowledge
Posts: 1082
Joined: Mon Oct 29, 2012 8:12 pm
Location: Vancouver Island
Contact: Website

Re: Extra 16M memory on Pi2/Pi3

Wed Apr 12, 2017 5:33 pm

Interesting idea, and I have a very faintly possible explanation why using 1024 instead of 1023 might cause an issue. Do remember that "very faintly possible" part.

This derives from experience in the Mac version of Squeak Smalltalk where graphics are generated and bitblt'd (remember, BitBLT was invented by Dan Ingalls for Smalltalk) to a bitmap buffer. If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have. Boom, as Ivanova likes to put it.

Maybe, just maybe, Kodi occasionally does something similar.
Making Smalltalk on ARM since 1986; making your Scratch better since 2012

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Wed Apr 12, 2017 5:43 pm

timrowledge wrote: If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have.
It could be something like that. In general if user code touches a word beyond the last page you allocated I'd expect linux to give you a page fault and kill your process (with or without the 1024M).
But there may be things in the kernel where permissions are more relaxed (e.g. using DMA hardware) where reading beyond a 4K page is tolerated but beyond 1024M may be more fatal.

Hopefully someone will test this and find an application that fails reliably. That would be useful in tracking it down.

cjan
Posts: 487
Joined: Sun May 06, 2012 12:00 am

Re: Extra 16M memory on Pi2/Pi3

Sat Apr 15, 2017 1:32 am

in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.

tvjon
Posts: 561
Joined: Mon Jan 07, 2013 9:11 am

Re: Extra 16M memory on Pi2/Pi3

Sat Apr 15, 2017 12:07 pm

"This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty."

A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

I've been experimenting with vlc recently, in particular using its mmal video output, so popped my pi3's µSD card into the 0.

Attached is the result of trying to stream from an ip camera.

I then remembered I'd changed config.txt to use your new extra ram setting, so I changed it to fixed gpu value. 32 megs always results in a seg' fault when vlc is started.

I tried other values up to 256 meg's, all of which allowed vlc to load fine, but always display a black screen where the video should be.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.

vlc does allocate a lot of buffers, so I'll rebuild it with fewer to see if that lets it show the screen properly.

pi0.jpg
pi0.jpg (60.31 KiB) Viewed 13230 times

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Sun Apr 16, 2017 11:34 am

cjan wrote:in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.
That won't make any difference for video playback in chromium.

For kodi, then sure, more gpu memory is useful (256MB is recommended for Pi2/Pi3).

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Sun Apr 16, 2017 11:37 am

tvjon wrote: A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.
Don't set total_mem=1024 on a Pi0/Pi1 - it makes no sense and it will be clamped to 512 anyway.
If you have trouble running vlc then create a new thread - it has nothing to do with this one.

zipplet
Posts: 13
Joined: Sun Dec 11, 2016 4:05 pm
Location: Tokyo, Japan
Contact: Website

Re: Extra 16M memory on Pi2/Pi3

Sat Jun 24, 2017 12:53 pm

I have been using this setting with retropie + the latest updates to Raspbian without issue so far at 1024 on a Pi 3B - I'll report back if I manage to get it to crash. Retropie is a good stress test with some of the emulators.

One other way to play with this would be to use that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel - then the theories about touching bad memory addresses could be proven/etc.

B.Goode
Posts: 4689
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Extra 16M memory on Pi2/Pi3

Sat Jun 24, 2017 1:16 pm

that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel
Ultibo?

https://ultibo.org

runboy93
Posts: 306
Joined: Tue Feb 28, 2017 1:17 pm
Location: Finland

Re: Extra 16M memory on Pi2/Pi3

Sat Jun 24, 2017 7:25 pm

768M arm, 256M gpu

config.txt should have "total_mem=1024" by default, or better informed, it came as little surprise for me too while ago.

16M may sound very little, but with RPi every bit is always a bonus.

That crash with kodi was maybe just very rare case it happened, atleast haven't seen myself nor on internet talking about crash on Raspbian because of this tweak.
Aluminum case for Raspberry Pi 3
https://goo.gl/3QRuVw

RPi 3 tweaks by runboy93
https://goo.gl/o8grny

runboy93
Posts: 306
Joined: Tue Feb 28, 2017 1:17 pm
Location: Finland

Re: Extra 16M memory on Pi2/Pi3

Fri Jul 28, 2017 8:13 pm

Interesting update on github, bootcode: total_mem=1024 by default.

https://github.com/raspberrypi/firmware ... 2c99fae437
Aluminum case for Raspberry Pi 3
https://goo.gl/3QRuVw

RPi 3 tweaks by runboy93
https://goo.gl/o8grny

cjan
Posts: 487
Joined: Sun May 06, 2012 12:00 am

Re: Extra 16M memory on Pi2/Pi3

Sat Jul 29, 2017 2:31 am

runboy93 wrote:
Fri Jul 28, 2017 8:13 pm
Interesting update on github, bootcode: total_mem=1024 by default.

https://github.com/raspberrypi/firmware ... 2c99fae437
so, no need total_mem=1024 in config.txt ?

beta-tester
Posts: 1153
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: Extra 16M memory on Pi2/Pi3

Sat Jul 29, 2017 6:26 am

dom wrote:
Wed Apr 12, 2017 4:51 pm
If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024
The extra 16M will be enabled.
why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?
{ I only give negative feedback }
RPi Model B (rev1, 256MB) & B (rev2, 512MB) & B+, RPi2B (1GB), 64GB microSDXC1 class 10, HDMI 1920x1080, keyboard-mouse-combo (wireless), PiCamera, ethernet-cable, 5V/1.2A power supply, Wifi dongle (rt5370)

bensimmo
Posts: 1928
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Extra 16M memory on Pi2/Pi3

Sat Jul 29, 2017 6:57 am

beta-tester wrote:
Sat Jul 29, 2017 6:26 am
dom wrote:
Wed Apr 12, 2017 4:51 pm
If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024
The extra 16M will be enabled.
why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?
Creating a github issue with the documentation tends to work well for these things.
If you follow the link on the page you end up here
https://github.com/raspberrypi/document ... /memory.md

Chances are this was announced for a bit of testing, it's not been used a lot, maybe the default now? And they just forgot and not been an issue.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18215
Joined: Sat Jul 30, 2011 7:41 pm

Re: Extra 16M memory on Pi2/Pi3

Sat Jul 29, 2017 7:44 am

bensimmo wrote:
Sat Jul 29, 2017 6:57 am
beta-tester wrote:
Sat Jul 29, 2017 6:26 am
dom wrote:
Wed Apr 12, 2017 4:51 pm
If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024
The extra 16M will be enabled.
why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?
Creating a github issue with the documentation tends to work well for these things.
If you follow the link on the page you end up here
https://github.com/raspberrypi/document ... /memory.md

Chances are this was announced for a bit of testing, it's not been used a lot, maybe the default now? And they just forgot and not been an issue.
Indeed, unless I'm told that a command has changed, its unlikely that the docs get updated. Because engineers.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Sun Jul 30, 2017 3:32 pm

cjan wrote:
Sat Jul 29, 2017 2:31 am
so, no need total_mem=1024 in config.txt ?
If you are running latest rpi-update firmware then no need for total_mem=1024 in config.txt. The extra 16M will be available.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Sun Jul 30, 2017 3:39 pm

beta-tester wrote:
Sat Jul 29, 2017 6:26 am
why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?
Some options are only used for testing and are not recommended for users outside of a specific testing thread (like this one).

I'd prefer not to put these test options into the official documentation as they are not generally useful.
Users who play with total_mem without following a thread like this are more than likely to break their system.

Now we have enabled the extra 16M by default there is no reason to use total_mem other than to make your Pi run less well
(you can't set it any higher than default and setting it lower will only reduce the amount of memory available).

runboy93
Posts: 306
Joined: Tue Feb 28, 2017 1:17 pm
Location: Finland

Re: Extra 16M memory on Pi2/Pi3

Sun Jul 30, 2017 8:09 pm

There were (don't know are there still) crash with this tweak on certain Kodi based systems, not sure how many happened with RPi 2/3, atleast none on my system.

Very happy enabled by default, less tweaking needed. Need update my guide for new image (when comes)

You can always check with commands on my guide how much memory for arm/gpu, I have arm 768 and gpu 256.
Aluminum case for Raspberry Pi 3
https://goo.gl/3QRuVw

RPi 3 tweaks by runboy93
https://goo.gl/o8grny

RonR
Posts: 209
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Extra 16M memory on Pi2/Pi3

Sun Jul 30, 2017 10:54 pm

On my two RPi-3's, with a default config.txt, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=948M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=76M

Total = 1024

After adding the following line to config.txt:

gpu_mem=16

and rebooting, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=998M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=16M

Total = 1014

Where's my other 10 MB?

User avatar
rpdom
Posts: 11865
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Extra 16M memory on Pi2/Pi3

Mon Jul 31, 2017 5:22 am

RonR wrote:
Sun Jul 30, 2017 10:54 pm
Where's my other 10 MB?
You'll never get the full 1024MB as the top part of the memory address space is used for memory mapped IO such as the GPIO and other peripherals.

RonR
Posts: 209
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Extra 16M memory on Pi2/Pi3

Mon Jul 31, 2017 5:42 am

rpdom wrote:
Mon Jul 31, 2017 5:22 am
You'll never get the full 1024MB as the top part of the memory address space is used for memory mapped IO such as the GPIO and other peripherals.
Huh?
RonR wrote:
Sun Jul 30, 2017 10:54 pm
On my two RPi-3's, with a default config.txt, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=948M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=76M

Total = 1024
runboy93 wrote:
Sun Jul 30, 2017 8:09 pm
I have arm 768 and gpu 256.

User avatar
rpdom
Posts: 11865
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Extra 16M memory on Pi2/Pi3

Mon Jul 31, 2017 6:38 am

Hmm, maybe the top bit is counted as GPIO memory now?

RonR
Posts: 209
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Extra 16M memory on Pi2/Pi3

Tue Aug 01, 2017 1:22 am

dom,

Any explanation why 10 MB is missing?

beta-tester
Posts: 1153
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: Extra 16M memory on Pi2/Pi3

Tue Aug 01, 2017 10:47 am

RonR wrote:
Tue Aug 01, 2017 1:22 am
Any explanation why 10 MB is missing?
maybe an allignement thing.
maybe the CPU/GPU memory location needs to be alligned to specific borders.
and if they are not alligned well, then there are unaccessable gaps in between the both memory allocations.

or the chunks for CPU allocation is different to the allocation chunks of the GPU.
or there is a hard coded min/max size you hit.
{ I only give negative feedback }
RPi Model B (rev1, 256MB) & B (rev2, 512MB) & B+, RPi2B (1GB), 64GB microSDXC1 class 10, HDMI 1920x1080, keyboard-mouse-combo (wireless), PiCamera, ethernet-cable, 5V/1.2A power supply, Wifi dongle (rt5370)

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5106
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Extra 16M memory on Pi2/Pi3

Tue Aug 01, 2017 3:23 pm

RonR wrote:
Sun Jul 30, 2017 10:54 pm
Where's my other 10 MB?
The gpu consists of reloc (relocatable heap) and non-reloc (code, data, malloc heap, logging).

reloc is used for larger buffers, used by (firmware side) 3D, video decode, video encode, camera, framebuffer etc.
reloc may be accessed by the arm when using zero-copy mode of mmal, or mailbox interface for allocating buffers (e.g. hevc, hello_fft).

As such we don't want reloc in the top 16M or the arm will fail if it want to access it.

Normally non-reloc is 20M, and the reloc will be (gpu_mem-20).
But when gpu_mem < 32 we switch to the cutdown firmware which has non-reloc = 6M (due to removing 3D/video code) and reloc=10M.

But with this layout the arm can't see the reloc as it's in the top 16M, so we move it down by 10M, hence the missing 10M you see.

Now many of the reasons for the arm seeing the reloc don't apply when using the cutdown firmware (no 3d, video etc).
But the framebuffer is still required and that is allocated from reloc, and the arm needs access.

Return to “Advanced users”

Who is online

Users browsing this forum: mattmiller and 16 guests