Unlinked files and seemingly empty full partitions

$ df -P /dev/sda3
Filesystem   1024-blocks   Used      Available   Capacity Mounted on
/dev/sda3        8256952   7007368   830160      90%      /var

$ df -i /dev/sda3
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda3             524288   21548  502740    5% /var

So far, so bad. The disk is full and we can’t blame it on inodes. Since it’s /var, it’s probably just some log files…

$ du -sh /var
1.9G    /var

…or not?

One of the features of Unix filesystems (compared to Windows) is that “deleting” a file does not necessary delete it. The file will be unlinked (so it doesn’t appear in a file system anymore), but still remains usable. Open file handles to this file are tracked, and the file will only be deleted when the last handle is closed (this is why Windows has the annoying “Cannot delete file, it’s open somewhere [and I won’t tell you where]” and Linux doesn’t). df obviously has to include unlinked files, but du doesn’t. For that, we have lsof:

When +L is followed by a number, only files having a link count less than that number will be listed. (No number may follow -L.) A specification of the form “+L1” will select open files that have been unlinked. A specification of the form “+aL1 ” will select unlinked open files on the specified file system.

$ lsof +L1
apache2  5102 www-data    2w   REG    8,3    627670     0 450637 /var/log/apache2/error.log.1 (deleted)
apache2  5102 www-data    7w   REG    8,3   2030629     0 450922 /var/log/apache2/other_vhosts_access.log.1 (deleted)
apache2  5102 www-data    8w   REG    8,3   2030629     0 450922 /var/log/apache2/other_vhosts_access.log.1 (deleted)
…and fifty other big log files…

It actually were log files to blame for this – Apache holds a handle to its closed log files after logrotate rotated (and deleted them). An Apache restart after logrotate runs fixes this particular problem.

(Thanks to ServerFault for pointing to lsof +L1!)