How to diagnose No space left on device error

[spikedog]

Hi,

I'm new to Linux and I've just set up a Raspberry Pi 2 with Raspbian Jessie.

The device runs fine for about a day then I start getting errors "No space left on device". I can't find which device is causing the problem and the errors come with a number of commands. A reboot fixes the problem for a while.

I've had this problem wile installing Apache and PHP using sudo apt-get and I often get it while starting and stopping services. This is my most recent error

Code: Select all

pi@pi1:~ $ sudo systemctl restart apache2.service
Error: No space left on device
pi@pi1:~ $ sudo service apache2 restart
Error: No space left on device
pi@pi1:~ $ sudo /etc/init.d/apache2 restart
[....] Restarting apache2 (via systemctl): apache2.serviceError: No space left on device
. ok

The problem isn't just with Apache2. I'm getting this with all services.

As far as I can tell I have space left on all devices

Code: Select all

pi@pi1:~ $ df -hl
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  3.8G  9.7G  29% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M   13M  451M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   60M   20M   41M  33% /boot
/dev/sda1        29G  7.7G   19G  29% /data0
tmpfs            93M     0   93M   0% /run/user/1000
pi@pi1:~ $ df -hil
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/root        923K  177K  747K   20% /
devtmpfs         115K   354  115K    1% /dev
tmpfs            116K     1  116K    1% /dev/shm
tmpfs            116K   548  116K    1% /run
tmpfs            116K     6  116K    1% /run/lock
tmpfs            116K    10  116K    1% /sys/fs/cgroup
/dev/mmcblk0p1      0     0     0     - /boot
/dev/sda1        1.8M   95K  1.8M    6% /data0
tmpfs            116K     5  116K    1% /run/user/1000

Any ideas how I can get more information on what is causing the error?

Thanks

[QuietZone]

This is usually caused by the combination of two erroneous actions:

1) Installing Raspbian from a "raw image" instead of from NOOBS.

2) Failing to "expand the rootfs" (using "raspi-config")

[kusti8]

This is usually caused by the combination of two erroneous actions:

1) Installing Raspbian from a "raw image" instead of from NOOBS.

2) Failing to "expand the rootfs" (using "raspi-config")

As shown in the df output the OP has done so.

[spikedog]

Thanks for the replies.

I have found the problem now and it isn't obvious from the error message.

I have enough physical space and already run raspi-config to use all available space.

The problem only occurs when using systemctl (either directly or during installation, reboot etc) and is caused by the max_user_watches limit. The default value of 8192 might be enough for most but I'm also running crashplan and the watch limit is being exhausted. This wasn't a problem on my previous installation using Wheezy, probably due to the introduction of systemd in Jessie.

My solution is to add

fs.inotify.max_user_watches=1048576

to /etc/sysctl.conf and the problem hasn't come back. I'll keep watching in case other problems arise.

[manderso]

Thanks for the replies.

I have found the problem now and it isn't obvious from the error message.

I have enough physical space and already run raspi-config to use all available space.

The problem only occurs when using systemctl (either directly or during installation, reboot etc) and is caused by the max_user_watches limit. The default value of 8192 might be enough for most but I'm also running crashplan and the watch limit is being exhausted. This wasn't a problem on my previous installation using Wheezy, probably due to the introduction of systemd in Jessie.

My solution is to add

fs.inotify.max_user_watches=1048576

to /etc/sysctl.conf and the problem hasn't come back. I'll keep watching in case other problems arise.

This was my problem exactly. I'm running Crashplan on my pi 3 and am trying to install the lets encrypt client, which kept bombing at the nginx installation. Your solution did the trick. How'd you find it?

[manderso]

This page has more information on how this is happening:
http://unix.stackexchange.com/questions ... it-reached

[govindg279]

I got the same error, but in my case the the root file system is 100% and raspi-config also failed.
pi@raspberrypi:~ $ df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.6G 3.6G 0 100% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 463M 0 463M 0% /dev/shm
tmpfs 463M 6.3M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 463M 0 463M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot
tmpfs 93M 0 93M 0% /run/user/1000

The below command worked for me:
sudo resize2fs /dev/mmcblk0p2

[DougieLawson]

That 4GB SDCard isn't big enough for regular Raspbian Jessie. You can fit Jessie-lite on a 4GB card, but for regular Jessie you need at least 8GB.