When df and du dont match.

So I had a weird issue the other day, just after I had created an extent and added it to a volume group to extend my plex disk (mostly for more scratch space for the transcoder to do its job).

Anyway, I got an alert in Zabbix that the disk was nearly full.

This was odd, since I had just increased the disk some 30+ GB's. A quick look at the server and sure enough df showed that it was in fact used

df -h

It looked something similar to this.

Notice how the main root mount point was showing high usage

Note: I fixed this before i wrote this post. So I artificially re-created the condition for the screenshots.

I have a small script that I often use. I call it treesize, but in reality its just du with some extra stuff and I often use it to find where storage is being eaten up.

Heres the script:

#/bin/sh
du -k --max-depth=1 | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

And here is what an output would look like.

Now its worth noting a few things at this point.

1. Plex uses cifs mounts setup via fstab for its actual library content.
2. The NAS that stores those mounts is backed up nightly.
3. Plex itself is a Virtual Machine and also backed up nightly.

As a result I will sometimes have plex oddly lose its cifs mounts.This is likely due to the VM being "stunned" during backup, but could also happen during the btrfs snapshot and backup done on the NAS. Frankly, I never had the time nor the inclination to look too far into it because simply updating the server nightly and rebooting it fixes the issue in 95% of the cases.

So i got to googling (well "duckduckgoing") and came across some of the standard advice such as check to make sure inodes arent exhausted or run lsof to make sure a deleted file wasnt held up by a running process. But then i got to this page about bind mounts and finally this one that really got me thinking.

I do have plex setup to record some over-the-air TV (OTA) stuff, mostly local news and kids shots. What would happen if plex lost its cifs shares inexplicably AND Plex did a ton of recording to that same path location before I had time to catch it?

So I stopped the plex service and unmounted all shares, then re-ran my du/treesize script.

Sure enough, some extra storage utilization appears. And it all appears in the very share that I used Plex to record over the air TV to.

Note: As stated, in this example I used fallocate to create some large files to use as examples. My actual directory looked something like this snap which I grabbed from backups.

Sure enough that's exactly what happened. So in this case, its worth looking at any mounts you may be using and making sure that you don't have some files inexplicably loaded up in folder locations you typically use to mount shares...

Or just rebuild the server, I am not your parent.