obolon13
Posts: 4
Joined: Mon Oct 21, 2013 7:41 pm

yet one more "no space left on device" issue

Mon Oct 21, 2013 7:51 pm

I faced the problem: when I try to copy file under pi user I'm getting "no space left on device" error.
Under root user (sudo) all is okay - see the output below.
Preliminary I've performed filesystem check on boot, inode table is also not fully popuated.
This issue stops my transmission from normal work, because it also is running as non-root user.

Code: Select all

[email protected] /var/lib/transmission-daemon/incomplete $ cp /mnt/NAS/Video/Thumbs.db .
cp: writing `./Thumbs.db': No space left on device
cp: failed to extend `./Thumbs.db': No space left on device
[email protected] /var/lib/transmission-daemon/incomplete $ ls -al
total 8
drwxrwxrwx 2 debian-transmission debian-transmission 4096 Oct 21 22:31 .
drwxrwxrwx 5 root                root                4096 Sep 12 23:26 ..
-rwxr-xr-x 1 pi                  pi                     0 Oct 21 22:31 Thumbs.db
[email protected] /var/lib/transmission-daemon/incomplete $ rm Thumbs.db
[email protected] /var/lib/transmission-daemon/incomplete $ ls -al
total 8
drwxrwxrwx 2 debian-transmission debian-transmission 4096 Oct 21 22:45 .
drwxrwxrwx 5 root                root                4096 Sep 12 23:26 ..
[email protected] /var/lib/transmission-daemon/incomplete $ sudo cp /mnt/NAS/Video/Thumbs.db .
[email protected] /var/lib/transmission-daemon/incomplete $ ls -al
total 40
drwxrwxrwx 2 debian-transmission debian-transmission  4096 Oct 21 22:45 .
drwxrwxrwx 5 root                root                 4096 Sep 12 23:26 ..
-rwxr-xr-x 1 root                root                31232 Oct 21 22:45 Thumbs.db
[email protected] /var/lib/transmission-daemon/incomplete $

User avatar
jojopi
Posts: 3192
Joined: Tue Oct 11, 2011 8:38 pm

Re: yet one more "no space left on device" issue

Mon Oct 21, 2013 8:59 pm

"Yet one more" suggests that you have seen or heard of this message in the past, so I am not sure what you are asking. It always means the same thing: the filesystem is (almost) full.

The reason it still works under sudo is that a number of blocks are reserved for the root user. You do not want to completely fill those, because important parts of the system may stop functioning.

Are you aware that transmission is using /var/lib, presumably your SD card, and not /mnt/NAS?

"df -m" will confirm the size and fullness of the root filesystem. Make sure you have selected the "Expand Filesystem" option in raspi-config, if you did not install using NOOBS. Try "ncdu -x /" if you are not sure where the space is taken up.

obolon13
Posts: 4
Joined: Mon Oct 21, 2013 7:41 pm

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 6:25 am

Well, you're right the topic was discussed but it was either in French - which I don't understand - or I hardly can find the root issue of the problem in discussions in English.
I've performed extend_rootfs from raspi-config but it has no effect because rootfs used all space on SD card (4GB) already.

Indeed my initial idea was that root filesystem (to which /var/lib belongs to) is full:

Code: Select all

[email protected] ~ $ df -m
Filesystem               1M-blocks   Used Available Use% Mounted on
rootfs                        3719   3531         0 100% /
/dev/root                     3719   3531         0 100% /
devtmpfs                       243      0       243   0% /dev
tmpfs                           49      1        49   1% /run
tmpfs                            5      0         5   0% /run/lock
tmpfs                           98      0        98   0% /run/shm
/dev/mmcblk0p1                  56     17        40  30% /boot
tmpfs                           98      0        98   0% /tmp
//192.168.1.200/Music       947643 826258    121385  88% /mnt/NAS/Music
//192.168.1.200/Misc        947643 826258    121385  88% /mnt/NAS/Misc
//192.168.1.200/Pictures    947643 826258    121385  88% /mnt/NAS/Pictures
//192.168.1.200/Soft        947643 826258    121385  88% /mnt/NAS/Soft
//192.168.1.200/Video       947643 826258    121385  88% /mnt/NAS/Video
But as I'm not so big guru of Linux I was confused with behaviour of cp command from my previous post performed under root user. Another issue which confused me is that logs in /var/log (for example, /var/log/daemon.log) still are populated with messages. Now I see that it's being written under root user so it also receives a reserved blocks fo root.

I tried to use ncdu as you suggested but I can't get an idea where is the space used is located, it shows in total ~1.5GB used, while df reports 100% usage:

Code: Select all

--- / -----------------------------------------------------------------------------------------------------------------------------------------------
    1.1GiB [##########] /usr
. 217.1MiB [#         ] /var
   65.5MiB [          ] /lib
   34.1MiB [          ] /opt
    6.1MiB [          ] /sbin
    5.2MiB [          ] /bin
.   4.1MiB [          ] /etc
    3.9MiB [          ] /home
!  16.0KiB [          ] /lost+found
    8.0KiB [          ] /mnt
e   4.0KiB [          ] /srv
e   4.0KiB [          ] /selinux
!   4.0KiB [          ] /root
e   4.0KiB [          ] /media
!   4.0KiB [          ] /.pulse
    4.0KiB [          ]  .pulse-cookie
>   0.0  B [          ] /tmp
>   0.0  B [          ] /sys
>   0.0  B [          ] /run
>   0.0  B [          ] /proc
>   0.0  B [          ] /dev
>   0.0  B [          ] /boot

User avatar
jojopi
Posts: 3192
Joined: Tue Oct 11, 2011 8:38 pm

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 7:06 am

The ncdu results look normal for a recent install. One possibility for the missing ~2GB is that it is in a directory you cannot access, such as /root. I should probably have suggested "sudo ncdu -x /" to eliminate this risk, but I was guessing the data would be in /var/lib/transmission-daemon, which you do have access to.

The other possibility is that you have deleted the data already. In Linux you can remove a file that is in use, such as something transmission is still seeding, or a log file that is still being written to, but the space will not then be freed until that process has closed the file or exited.

obolon13
Posts: 4
Joined: Mon Oct 21, 2013 7:41 pm

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 12:48 pm

Same result of "sudo ncdu -x /": ~1.4GB of / is used.
May be the issue is with filesystem.
I performed

Code: Select all

[email protected] /var/log $ ls /f*
ls: cannot access /f*: No such file or directory
[email protected] /var/log $ sudo ls -al /f*
ls: cannot access /f*: No such file or directory
[email protected] /var/log $ sudo touch /forcefsck
[email protected] /var/log $ sudo ls -al /f*
-rw-r--r-- 1 root root 0 Oct 22 15:36 /forcefsck
[email protected] /var/log $
and then "sudo reboot" - same output.

Inode table is not full:

Code: Select all

[email protected] ~ $ sudo df -i
Filesystem                 Inodes IUsed    IFree IUse% Mounted on
rootfs                     245760 68392   177368   28% /
/dev/root                  245760 68392   177368   28% /
devtmpfs                    62205   315    61890    1% /dev
tmpfs                       62220   229    61991    1% /run
tmpfs                       62220     1    62219    1% /run/lock
tmpfs                       62220     2    62218    1% /run/shm
/dev/mmcblk0p1                  0     0        0     - /boot
tmpfs                       62220     4    62216    1% /tmp
//192.168.1.200/Music    30474240 84702 30389538    1% /mnt/NAS/Music
//192.168.1.200/Misc     30474240 84702 30389538    1% /mnt/NAS/Misc
//192.168.1.200/Pictures 30474240 84702 30389538    1% /mnt/NAS/Pictures
//192.168.1.200/Soft     30474240 84702 30389538    1% /mnt/NAS/Soft
//192.168.1.200/Video    30474240 84702 30389538    1% /mnt/NAS/Video
[email protected] ~ $
For me it seems to be some issue with filesystem.
Interesting that all worked okay until one day. What could cause the bug - power switch off: my Pi is connected to electricity outlet which may suffer from random power swithcing.
I wonder is there any way how to find a bug or I should re-install Pi from the scratch.

User avatar
jojopi
Posts: 3192
Joined: Tue Oct 11, 2011 8:38 pm

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 3:00 pm

Now I am sure that the phantom data is somewhere under /mnt/NAS. You cannot find it because the remote filesystems are mounted on top. Try running the ncdu with those other filesystems unmounted.

User avatar
FTrevorGowen
Forum Moderator
Forum Moderator
Posts: 5551
Joined: Mon Mar 04, 2013 6:12 pm
Location: Bristol, U.K.
Contact: Website

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 3:43 pm

jojopi wrote:Now I am sure that the phantom data is somewhere under /mnt/NAS. You cannot find it because the remote filesystems are mounted on top. Try running the ncdu with those other filesystems unmounted.
"ncdu" is new to me - I'll have to give it a go sometime. My usual method of finding "large amounts of stuff" during "houskeeping" is to run an alias to this:
"du -k | sort -n -r | head"
This (eventually) shows the "top ten" largest directories. (@obolon13 NB: if you still have the remote filesystems mounted when trying the above out the "sort" part could take a very long time)
Trev.
Still running Raspbian Jessie or Stretch on some older Pi's (an A, B1, 2xB2, B+, P2B, 3xP0, P0W, 2xP3A+, P3B+, P3B, B+, and a A+) but Buster on the P4B's. See: https://www.cpmspectrepi.uk/raspberry_pi/raspiidx.htm

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 3:53 pm

FTrevorGowen wrote:
jojopi wrote:Now I am sure that the phantom data is somewhere under /mnt/NAS. You cannot find it because the remote filesystems are mounted on top. Try running the ncdu with those other filesystems unmounted.
"ncdu" is new to me - I'll have to give it a go sometime. My usual method of finding "large amounts of stuff" during "houskeeping" is to run an alias to this:
"du -k | sort -n -r | head"
This (eventually) shows the "top ten" largest directories. (@obolon13 NB: if you still have the remote filesystems mounted when trying the above out the "sort" part could take a very long time)
Trev.
...
I used to use something like
du -k / | sort -nro /tmp/filesizes.nro
but they can get large ;-p
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

obolon13
Posts: 4
Joined: Mon Oct 21, 2013 7:41 pm

Re: yet one more "no space left on device" issue

Tue Oct 22, 2013 7:07 pm

bingo!
jojopi you're right - there is a large file in /mnt/NAS/Video.
I've unmounted remoute filesystems from /mnt/NAS:

Code: Select all

[email protected] ~ $ mount
/dev/root on / type ext4 (rw,noatime,user_xattr,barrier=1,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=248820k,nr_inodes=62205,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=49780k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=99540k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=99540k)
and here is an output of "ncdu -x /":

Code: Select all

- / ------------------------------------------------------------------------------------------------------------------------------------------------
    2.0GiB [##########] /mnt
    1.1GiB [#####     ] /usr
. 217.5MiB [#         ] /var
   65.5MiB [          ] /lib
   34.1MiB [          ] /opt
    6.1MiB [          ] /sbin
    5.2MiB [          ] /bin
.   4.1MiB [          ] /etc
    3.9MiB [          ] /home
!  16.0KiB [          ] /lost+found
e   4.0KiB [          ] /srv
e   4.0KiB [          ] /selinux
!   4.0KiB [          ] /root
e   4.0KiB [          ] /media
!   4.0KiB [          ] /.pulse
    4.0KiB [          ]  .pulse-cookie
>   0.0  B [          ] /tmp
>   0.0  B [          ] /sys
>   0.0  B [          ] /run
>   0.0  B [          ] /proc
>   0.0  B [          ] /dev
>   0.0  B [          ] /boot
Here is a picture of used blocks after deletion of that file:

Code: Select all

[email protected] ~ $ df -m
Filesystem               1M-blocks   Used Available Use% Mounted on
rootfs                        3719   1477      2054  42% /
/dev/root                     3719   1477      2054  42% /
devtmpfs                       243      0       243   0% /dev
tmpfs                           49      1        49   1% /run
tmpfs                            5      0         5   0% /run/lock
tmpfs                           98      0        98   0% /run/shm
/dev/mmcblk0p1                  56     17        40  30% /boot
tmpfs                           98      0        98   0% /tmp
//192.168.1.200/Music       947643 826258    121385  88% /mnt/NAS/Music
//192.168.1.200/Misc        947643 826258    121385  88% /mnt/NAS/Misc
//192.168.1.200/Pictures    947643 826258    121385  88% /mnt/NAS/Pictures
//192.168.1.200/Soft        947643 826258    121385  88% /mnt/NAS/Soft
//192.168.1.200/Video       947643 826258    121385  88% /mnt/NAS/Video
I guess the issue happened when my remote NAS was unmounted (also because power switch) but transmission have downloaded large file to rootfs directory of rather than remote NAS.

jojopi, thank you very much for help!

Hellwood
Posts: 2
Joined: Tue Nov 11, 2014 11:57 am

Re: yet one more "no space left on device" issue

Tue Nov 11, 2014 12:06 pm

Hello community.

I have exactly the same problem.
I have reboot my server many time and the problem returns.
And I confirm is when I adding torrents, rootfs and / dev / root fill 100%


I look everywhere but I can not find a solution, and it is the only subject that points perfectly the problem.

My files are received in the correct folder and file .part are also visible.

I'm french so please excuse my bad vocabulary (is google translation fault)

Hellwood
Posts: 2
Joined: Tue Nov 11, 2014 11:57 am

Re: yet one more "no space left on device" issue

Wed Nov 12, 2014 1:27 pm

My bad,

I m not very good in linux, so I have create a download folder directly in / and not in /home ....

So is my fault.


Sorry

User avatar
dentex
Posts: 180
Joined: Sun May 20, 2012 7:58 am
Location: Italy
Contact: Website

Re: yet one more "no space left on device" issue

Tue Jan 20, 2015 12:48 pm

I would like to say a big THANK YOU !!! , for the huge HELP received through this thread.
I discovered the awesome ncdu tool and found where the problem was: up to 2,8 Gb of logs into /var/log.

Ciao!
One of my projects with the Raspberry Pi:
--------------------------------------------------------
(Raspberry) Pi Aquarium Controller: https://github.com/dentex/piac

Return to “Troubleshooting”