## No space left on device

The bulk of this post comes from this excellent piece No space left on device – running out of Inodes by Ivan Kuznetsov.

Recently I tried to upate/upgrade a Linux machine, but failed. I got the notice that there were no space left on it. I saw that this was not true but had no time to deal with this. I saw that the machine was working and left it at that.

Next time this happened was as I was going to update the installed gems on that machine. This was the result

user@remote:~$sudo gem update Updating installed gems Updating addressable Fetching: addressable-2.5.2.gem (100%) Successfully installed addressable-2.5.2 Parsing documentation for addressable-2.5.2 Installing ri documentation for addressable-2.5.2 Installing darkfish documentation for addressable-2.5.2 Done installing documentation for addressable after 1 seconds Parsing documentation for addressable-2.5.2 Done installing documentation for addressable after 0 seconds Updating asciidoctor Fetching: asciidoctor-1.5.6.1.gem (100%) Successfully installed asciidoctor-1.5.6.1 Parsing documentation for asciidoctor-1.5.6.1 Installing ri documentation for asciidoctor-1.5.6.1 ERROR: While executing gem ... (Errno::ENOSPC) No space left on device @ rb_sysopen - ./Asciidoctor/Table/Cell/cdesc-Cell.ri  Issuing df -h gave me no clue. user@remote:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            487M     0  487M   0% /dev
tmpfs           100M   12M   88M  12% /run
/dev/xvda1      7.8G  5.7G  1.7G  78% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1000


Searching for the error message led me to Ivan Kuznetsov’s post, where I learned about the limited number of inodes leading to this type of error. He suggests diagnosing it by using the -i option to df. Which confirmed the problem.

user@remote:~# df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           124458    360 124098    1% /dev
tmpfs          126778    508 126270    1% /run
/dev/xvda1     524288 521615   2673  100% /
tmpfs          126778      1 126777    1% /dev/shm
tmpfs          126778      8 126770    1% /run/lock
tmpfs          126778     16 126762    1% /sys/fs/cgroup
tmpfs          126778      4 126774    1% /run/user/1000


Ivan then suggest to run a command that counts the underlying inodes of each top directory on the machine. The output from this is quite lengthy, so here some of it is:

user@remote:~# sudo for i in /*; do echo $i; find$i |wc -l; done
/bin
173
...
/etc
2509
/home
52
...
/proc
53120
...
/usr
480921


The almost half a million files in /usr looked a little hefty and modyfing the for loop gave the following result:

user@remote:~# sudo for i in /usr/*; do echo $i; find$i |wc -l; done
/usr/bin
995
/usr/games
1
/usr/include
2143
/usr/lib
10509
/usr/local
40
/usr/sbin
201
/usr/share
64089
/usr/src
400589


/usr/src with 400589 files and directories looked bloated. In there I found sources and headers of the kernels ever installed on the machine.

user@remote:~# ls /usr/src/


Of course, of these, only the latest version is needed. So I cut-n-pasted the output of the ls command.

user@remote:/usr/src# sudo rm -Rf linux-headers-4.4.0-79 linux-headers-4.4.0-89 \


This solved the issue with the depleeted inodes and as a new df -i gave me

user@remote:/usr/src# df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           124458    360 124098    1% /dev
tmpfs          126778    508 126270    1% /run
/dev/xvda1     524288 147778 376510   29% /
tmpfs          126778      1 126777    1% /dev/shm
tmpfs          126778      8 126770    1% /run/lock
tmpfs          126778     16 126762    1% /sys/fs/cgroup
tmpfs          126778      4 126774    1% /run/user/1000


After this I could update/upgrade Ubuntu as well as the gems with no problem.

Great thanks to Ivan Kuznetsov for the piece No space left on device – running out of Inodes which ultimately saved my day – perhaps the machine as well.