Ignore Your Inodes at Your Own Peril!

Who knew that ignoring the status of your inodes could bring down your Linux server?

I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my Linux server.

First though, just what is an inode? It’s an object used by Unix systems to identify the disk location and attributes of files within a filesystem. There will usually be exactly one inode for each file or directory (the exception being hard linked files that can share a single inode).

You can see the inode numbers for each of the objects within a directory by running ls -i 

$ ls -i
59776347 AsciidocFX
55312398 Desktop
55312402 Documents
55312399 Downloads
55312390 examples.desktop
56628996 git
55312403 Music
55312404 Pictures
55312401 Public
55312405 Videos
56236523 VirtualBox VMs
$

Now here’s the tricky bit. Just like the available disk space determines the upper limit on the files you can create or download to a computer, the number of files you can have is similarly capped by the an inode limit. To see your limit, run df -i and check out the value for your root directory under the Inodes column

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 60547072 628017 59919055 2% /
$

In the case of my home workstation, I’ve got more than 60 million inodes in total, of which only 2% are being used. No trouble there.

But when I logged into my server to run a security update the other day, I was hit with this:

dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-100_3.13.0-100.147_all.deb (--unpack):
unable to create `/usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig'): No space left on device

Note the “No space left on device” warning. No space? I ran df and saw that there were still plenty of free GBs on hand. Just to be sure, I tried to create a new file in my home directory (to rule out some kind of permissions problem). No luck there:

$ touch ~/newfile
touch: cannot touch ‘/home/ubuntu/newfile’: No space left on device

I’ll admit that I didn’t immediately think to look at the inodes – as I said above: who knew…? But a quick Google search revealed that I wasn’t the first guy to get hit with this. Suitably informed (and embarrassed) I remembered that df -i will display file system inode usage, and here’s what it looked like for me:

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 524288 0 100% /

IFree = 0, IUsed% = 100%. Not good. But what could possibly be generating enough files to drain my inode supply? Tracking down the directories containing the most files (i.e., using up the most inodes) can be done using some variation of this find command run from your root directory:

$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

But there’s a catch. Find saves its raw data in a temporary file…but it can’t do that as long as you have no free inodes. So you’ll have to delete one or two unneeded files just to get yourself going.

Read more...