Raspbian refuses to use Swap Space

[Kradenko]

Hi all,

I have been using the new Raspbian Buster OS on a Raspberry Pi 4 with 4GB of memory. I have also added a 750GB external HDD to the PI and placed a 8GB swap file on this external harddrive. Below is my configuration as per multiple tutorials and posts online.

Code: Select all

# /etc/dphys-swapfile - user settings for dphys-swapfile package
# author Neil Franklin, last modification 2010.05.05
# copyright ETH Zuerich Physics Departement
#   use under either modified/non-advertising BSD or GPL license

# this file is sourced with . so full normal sh syntax applies

# the default settings are added as commented out CONF_*=* lines


# where we want the swapfile to be, this is the default
CONF_SWAPFILE=/mnt/external/swapfile

# set size to absolute value, leaving empty (default) then uses computed value
#   you most likely don't want this, unless you have an special disk situation
CONF_SWAPSIZE=8192

# set size to computed value, this times RAM size, dynamically adapts,
#   guarantees that there is enough swap without wasting disk space on excess
#CONF_SWAPFACTOR=2

# restrict size (computed and absolute!) to maximally this limit
#   can be set to empty for no limit, but beware of filled partitions!
#   this is/was a (outdated?) 32bit kernel limit (in MBytes), do not overrun it
#   but is also sensible on 64bit to prevent filling /var or even / partition
CONF_MAXSWAP=8192

Code: Select all

[email protected]:~/src $ free -h
              total        used        free      shared  buff/cache   available
Mem:          3.9Gi       235Mi       377Mi       147Mi       3.3Gi       3.4Gi
Swap:         8.0Gi       3.0Mi       8.0Gi

Also when I look at "top" it doesn't ever go into this swap space.

Code: Select all

top - 17:49:01 up  1:44,  3 users,  load average: 0.04, 0.68, 2.49
Tasks: 134 total,   1 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  1.0 us,  1.9 sy,  0.0 ni, 97.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3955.6 total,    377.4 free,    235.1 used,   3343.1 buff/cache
MiB Swap:   8192.0 total,   8188.5 free,      3.5 used.   3431.3 avail Mem

All processing and large compilations fail with a "out of memory error" but it never uses the swap space. Does anyone have an idea why it won't allow me to use any of the swap space?

Any suggestions to resolve this will be much appreciated.

[jj_0]

Raspbian uses a different mechanism for configuring the swapfile.

[Kradenko]

Raspbian uses a different mechanism for configuring the swapfile.

Tried that. It allocates the disk space and shows it as swap space. But never uses it. Wondering if it's maybe a kernel or software related issue?

[jj_0]

Hmmm. Can you show the output of:

Code: Select all

 systemctl status dphys-swapfile

Maybe you need to remove the entries you put in fstab?

[Kradenko]

Hmmm. Can you show the output of:

Code: Select all

 systemctl status dphys-swapfile

Maybe you need to remove the entries you put in fstab?

Here is the output:

Code: Select all

[email protected]:~ $  systemctl status dphys-swapfile
● dphys-swapfile.service - dphys-swapfile - set up, mount/unmount, and delete a swap file
   Loaded: loaded (/lib/systemd/system/dphys-swapfile.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2019-10-01 16:04:00 BST; 3h 58min ago
     Docs: man:dphys-swapfile(8)
  Process: 332 ExecStart=/sbin/dphys-swapfile setup (code=exited, status=0/SUCCESS)
  Process: 410 ExecStart=/sbin/dphys-swapfile swapon (code=exited, status=0/SUCCESS)
 Main PID: 410 (code=exited, status=0/SUCCESS)

Oct 01 16:03:59 raspberrypi systemd[1]: Starting dphys-swapfile - set up, mount/unmount, and delete a swap file...
Oct 01 16:03:59 raspberrypi dphys-swapfile[332]: want /mnt/external/swapfile=8192MByte, checking existing: deleting wrong size file (2147483648), generating swapfile ... of 8192MBytes
Oct 01 16:04:00 raspberrypi systemd[1]: Started dphys-swapfile - set up, mount/unmount, and delete a swap file.

Also, here is my fstab:

Code: Select all

[email protected]:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=4710f635-01  /boot           vfat    defaults          0       2
PARTUUID=4710f635-02  /               ext4    defaults,noatime  0       1
/dev/sda1       /mnt/external   ext4    auto,nofail,user,noatime        0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

[jj_0]

Sorry, that's as far as I can imagine.

[logcabin]

Maybe the swap file is too large. For test purposes, try reducing the size to 2 GB.

[LTolledo]

depending on how you "load-up" applications on your RPi4B-4G it may never use the swap file if usage is "normal desktop use" as there are plenty of RAM for the system to "play around"
that's why the it was recommended before to keep the swap-file deactivated.

you can verify its use, as well as the RAM use by running

Code: Select all

htop

is there a special reason why you need to enable swap on your system?

[Kradenko]

depending on how you "load-up" applications on your RPi4B-4G it may never use the swap file if usage is "normal desktop use" as there are plenty of RAM for the system to "play around"
that's why the it was recommended before to keep the swap-file deactivated.

you can verify its use, as well as the RAM use by running

Code: Select all

htop

is there a special reason why you need to enable swap on your system?

htop is an awesome tool! Wow, didn't know it existed. Thnx!

There are two things I am trying to do on the Raspberry Pi.

1. I am trying to import map data into a postgres database. Here is the writeup about it https://switch2osm.org/manually-buildin ... 18-04-lts/. It am not trying to import the entire globe at least. Just a medium size area.
2. I am trying to generate the data files for the open source routing machine. This can be found here https://github.com/Project-OSRM/osrm-ba ... nning-OSRM.

So it's not day to day type activities. It is big data crunching. Really strange that even though the swap space is active, it just refuses to use the swap space.

[Kradenko]

Just to add to this. Is there something Rasbian that prevents apps from using more than X amount of ram? maybe a page limit of sorts?

I tried running it now again and it bombed out with the following output:

Code: Select all

[email protected]:/mnt/external/maps $ osrm-extract -p ~/osrm-backend/profiles/car.lua africa-latest.osm.pbf
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /home/pi/osrm-backend/profiles/car.lua
[info] Input file: africa-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by osmium/1.8.0
[info] timestamp: 2019-09-30T20:18:02Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] RAM: peak bytes used: 1193299968
[error] [exception] std::bad_alloc
[error] Please provide more memory or consider using a larger swapfile

That says that it uses no more than 1193,3 MB. Could this be a software issue? Just strange that all these functions bomb out on such low consumption.

Thanks again for the assistance!

[laurent]

Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

[Kradenko]

Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

I have two ways I guess I could try see what causes it.

1. Try the new 64Bit Kernel for Raspbian
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

Will try tonight and report back.

[Kradenko]

Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

I have two ways I guess I could try see what causes it.

1. Try the new 64Bit Kernel for Raspbian
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

Will try tonight and report back.

Ok, so setting the kernel to 64Bit made no difference. As for the VM test on Ubuntu it worked flawlessly. It allocated swap memory as needed on exactly the same specs as the Pi 4 with 4GB of RAM and 8GB of swap.

memory-2.jpg (150.19 KiB) Viewed 1209 times

What I did notice while watching the htop as LTolledo had suggested, was the VIRT column would not allocate more that 1280MB of memory. Then it runs out of memory. So I am wondering now if there is a limit set by the OS which doesn't allow more than 1280MB of memory to be allocated to a process? Could this be?

[dickon]

2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

x86, or amd64? If the latter, then you're hitting per-process limits on the Pi; given it's a database, it may not be actually using the memory, but may be using the address space.

I'd be tempted to boot the 64b kernel with a 64b Debian userland (in a chroot is fine) and see if that fixes it; I suspect it will. I think you've hit one of those niche use-cases where a full 64b OS is useful.

[Kradenko]

2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

x86, or amd64? If the latter, then you're hitting per-process limits on the Pi; given it's a database, it may not be actually using the memory, but may be using the address space.

I'd be tempted to boot the 64b kernel with a 64b Debian userland (in a chroot is fine) and see if that fixes it; I suspect it will. I think you've hit one of those niche use-cases where a full 64b OS is useful.

The Ubuntu in the VM that I tested is amd64. So you might be right there. Will give it a go and see. Thanks!

[tului]

What's the memory split on 32 bit ARMfor apps/kernel

[Kradenko]

Well, I have now installed Gentoo 64Bit on the Pi and it has gone a lot better. I can now actually use the full 4GB of memory. It also allows me to go deep into swap space. 8GB on an external harddrive. Still bombs on larger datasets.

Will wait for the official 64Bit launch of Raspbian and try again.

Thanks for the assistance!

[tului]

Well, I have now installed Gentoo 64Bit on the Pi and it has gone a lot better. I can now actually use the full 4GB of memory. It also allows me to go deep into swap space. 8GB on an external harddrive. Still bombs on larger datasets.

Will wait for the official 64Bit launch of Raspbian and try again.

Thanks for the assistance!

Afaik it's official just in testing.