The bulk of this post comes from this excellent piece No space left on device – running out of Inodes by Ivan Kuznetsov.
Recently I tried to upate/upgrade a Linux machine, but failed. I got the notice that there were no space left on it. I saw that this was not true but had no time to deal with this. I saw that the machine was working and left it at that.
Next time this happened was as I was going to update the installed gems on that machine. This was the result
user@remote:~$ sudo gem update
Updating installed gems
Updating addressable
Fetching: addressable-2.5.2.gem (100%)
Successfully installed addressable-2.5.2
Parsing documentation for addressable-2.5.2
Installing ri documentation for addressable-2.5.2
Installing darkfish documentation for addressable-2.5.2
Done installing documentation for addressable after 1 seconds
Parsing documentation for addressable-2.5.2
Done installing documentation for addressable after 0 seconds
Updating asciidoctor
Fetching: asciidoctor-1.5.6.1.gem (100%)
Successfully installed asciidoctor-1.5.6.1
Parsing documentation for asciidoctor-1.5.6.1
Installing ri documentation for asciidoctor-1.5.6.1
ERROR: While executing gem ... (Errno::ENOSPC)
No space left on device @ rb_sysopen - ./Asciidoctor/Table/Cell/cdesc-Cell.ri
Issuing df -h gave me no clue.
user@remote:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 12M 88M 12% /run
/dev/xvda1 7.8G 5.7G 1.7G 78% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/1000
Searching for the error message led me to Ivan Kuznetsov’s post, where
I learned about the limited number of inodes leading to this type of
error. He suggests diagnosing it by using the -i option to
df. Which confirmed the problem.
user@remote:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 124458 360 124098 1% /dev
tmpfs 126778 508 126270 1% /run
/dev/xvda1 524288 521615 2673 100% /
tmpfs 126778 1 126777 1% /dev/shm
tmpfs 126778 8 126770 1% /run/lock
tmpfs 126778 16 126762 1% /sys/fs/cgroup
tmpfs 126778 4 126774 1% /run/user/1000
Ivan then suggest to run a command that counts the underlying inodes of each top directory on the machine. The output from this is quite lengthy, so here some of it is:
user@remote:~# sudo for i in /*; do echo $i; find $i |wc -l; done
/bin
173
...
/etc
2509
/home
52
...
/proc
53120
...
/usr
480921
The almost half a million files in /usr looked a little hefty and
modyfing the for loop gave the following result:
user@remote:~# sudo for i in /usr/*; do echo $i; find $i |wc -l; done
/usr/bin
995
/usr/games
1
/usr/include
2143
/usr/lib
10509
/usr/local
40
/usr/sbin
201
/usr/share
64089
/usr/src
400589
/usr/src with 400589 files and directories looked bloated. In there
I found sources and headers of the kernels ever installed on the
machine.
user@remote:~# ls /usr/src/
linux-headers-4.4.0-101 linux-headers-4.4.0-79 linux-headers-4.4.0-89 linux-headers-4.4.0-96
linux-headers-4.4.0-101-generic linux-headers-4.4.0-79-generic linux-headers-4.4.0-89-generic linux-headers-4.4.0-96-generic
linux-headers-4.4.0-71 linux-headers-4.4.0-81 linux-headers-4.4.0-91 linux-headers-4.4.0-97
linux-headers-4.4.0-71-generic linux-headers-4.4.0-81-generic linux-headers-4.4.0-91-generic linux-headers-4.4.0-97-generic
linux-headers-4.4.0-75 linux-headers-4.4.0-83 linux-headers-4.4.0-92 linux-headers-4.4.0-98
linux-headers-4.4.0-75-generic linux-headers-4.4.0-83-generic linux-headers-4.4.0-92-generic linux-headers-4.4.0-98-generic
linux-headers-4.4.0-78 linux-headers-4.4.0-87 linux-headers-4.4.0-93
linux-headers-4.4.0-78-generic linux-headers-4.4.0-87-generic linux-headers-4.4.0-93-generic
Of course, of these, only the latest version is needed. So I
cut-n-pasted the output of the ls command.
user@remote:/usr/src# sudo rm -Rf linux-headers-4.4.0-79 linux-headers-4.4.0-89 \
linux-headers-4.4.0-96 linux-headers-4.4.0-79-generic \
linux-headers-4.4.0-89-generic linux-headers-4.4.0-96-generic \
linux-headers-4.4.0-71 linux-headers-4.4.0-81 linux-headers-4.4.0-91 \
linux-headers-4.4.0-97 linux-headers-4.4.0-71-generic \
linux-headers-4.4.0-81-generic linux-headers-4.4.0-91-generic \
linux-headers-4.4.0-97-generic linux-headers-4.4.0-75 \
linux-headers-4.4.0-83 linux-headers-4.4.0-92 linux-headers-4.4.0-98 \
linux-headers-4.4.0-75-generic linux-headers-4.4.0-83-generic \
linux-headers-4.4.0-92-generic linux-headers-4.4.0-98-generic \
linux-headers-4.4.0-78 linux-headers-4.4.0-87 linux-headers-4.4.0-93 \
linux-headers-4.4.0-78-generic linux-headers-4.4.0-87-generic \
linux-headers-4.4.0-93-generic
This solved the issue with the depleeted inodes and as a new df -i
gave me
user@remote:/usr/src# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 124458 360 124098 1% /dev
tmpfs 126778 508 126270 1% /run
/dev/xvda1 524288 147778 376510 29% /
tmpfs 126778 1 126777 1% /dev/shm
tmpfs 126778 8 126770 1% /run/lock
tmpfs 126778 16 126762 1% /sys/fs/cgroup
tmpfs 126778 4 126774 1% /run/user/1000
After this I could update/upgrade Ubuntu as well as the gems with no problem.
Great thanks to Ivan Kuznetsov for the piece No space left on device – running out of Inodes which ultimately saved my day – perhaps the machine as well.