When df and du dont match.

So I had a weird issue the other day, just after I had created an extent and added it to a volume group to extend my plex disk (mostly for more scratch space for the transcoder to do its job).

Anyway, I got an alert in Zabbix that the disk was nearly full.disk was near full

This was odd, since I had just increased the disk some 30+ GB's. A quick look at the server and sure enough df showed that it was in fact used

df -h

It looked something similar to this. df -h

Notice how the main root mount point was showing high usage

Note: I fixed this before i wrote this post. So I artificially re-created the condition for the screenshots.

I have a small script that I often use. I call it treesize, but in reality its just du with some extra stuff and I often use it to find where storage is being eaten up.

Heres the script:

#/bin/sh
du -k --max-depth=1 | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

And here is what an output would look like.treesize

Now its worth noting a few things at this point.

  1. Plex uses cifs mounts setup via fstab for its actual library content.
  2. The NAS that stores those mounts is backed up nightly.
  3. Plex itself is a Virtual Machine and also backed up nightly.

As a result I will sometimes have plex oddly lose its cifs mounts.This is likely due to the VM being "stunned" during backup, but could also happen during the btrfs snapshot and backup done on the NAS. Frankly, I never had the time nor the inclination to look too far into it because simply updating the server nightly and rebooting it fixes the issue in 95% of the cases.

So i got to googling (well "duckduckgoing") and came across some of the standard advice such as check to make sure inodes arent exhausted or run lsof to make sure a deleted file wasnt held up by a running process. But then i got to this page about bind mounts and finally this one that really got me thinking.

I do have plex setup to record some over-the-air TV (OTA) stuff, mostly local news and kids shots. What would happen if plex lost its cifs shares inexplicably AND Plex did a ton of recording to that same path location before I had time to catch it?

So I stopped the plex service and unmounted all shares, then re-ran my du/treesize script.

enter image description here

Sure enough, some extra storage utilization appears. And it all appears in the very share that I used Plex to record over the air TV to. enter image description here

Note: As stated, in this example I used fallocate to create some large files to use as examples. My actual directory looked something like this snap which I grabbed from backups.enter image description here

Sure enough that's exactly what happened. So in this case, its worth looking at any mounts you may be using and making sure that you don't have some files inexplicably loaded up in folder locations you typically use to mount shares...

Or just rebuild the server, I am not your parent.

How I cut the chord and built my own DVR

This will give a high level overview of how I canceled my cable subscription and rolled my own whole home DVR. I wont get too in the weeds about specifics because, in reality, I have a lot of existing "infrastructure" probably uncommon in most households.

Ultimately I grew tired of haggling with providers to get the same rates they happily give people with not billing history, and ultimately I realized we really only watch stuff that is basically free with a few exceptions.

To get started I needed some basics.

  1. Storage - A NAS, Desktop, heck even a small RaspberryPi COULD work (though i probably wouldnt reccomend it). For me this is a Synology Diskstation 1815+.
  2. Plex Server - A computer with a decent CPU (or even better one that supports Intel QuickSync). This can be a Windows Desktop, Linux, even Mac. It just needs to be running all the time. For me a built a virtual machine that runs on a Dell 7050 SFF
  3. Plex Pass - This is needed to stream and record OTA tv. You can start small at $5/mo and then just get the lifetime if everything works. A lifetime pass was less than 1 month of cable service..
  4. An Antenna - Or in my case, I'm lazy and went with not one, but two indoor models.
  5. A Tuner to convert the Antenna signal something the plex server can talk to.
  6. Rokus for the TVs - AppleTV's, or Firesticks anything that can run Plex will do. I went with Roku because I had one already, had Recently bought a cheap Roku TV AND most importantly, they work with universal remotes.
  7. Universal Remote - Simple and consistent across all TV's in the house. So guests can generally operate it.

Update - I recently move to these remotes for my rokus that were NOT roku TV (TCL brand). They are still consistent and didnt confuse some visitors as much.

Its worth noting you can combine items 1 and 2, especially if this is a new foray and you are buying hardware anyway. Plex has a good document on what NAS models are supported and can do transcoding. And again, you could get any old desktop, even off amazon and install plex and pop some hard drives in. The only real point is you need both something to store files with some bigger drives and something to stream with.

Once you have the hardware the fun begins.

First I setup file shares on the NAS, along with an account with access. The account was simply "plexserver" and it has modify rights to the file shares.

For all intents and purposes I have several shares

  1. DVR for shows I record.
  2. Television share for shows I own and just want to save.
  3. Movies
  4. Kids movies - I dont really want bambi cluttering up my adult movies.

Once I had the shares setup. I spun up linux server in my VMware lab.

I created the folder /plex and then a folder for each share in /plex.

I mounted the shares in /etc/crontab enter image description here

I used a config file in /etc/ to handle the credentials enter image description here Its very simple with only 2 lines. Make sure you set permissions right. enter image description here.

From there install plex, setup your libraries and you are off to the races.

Setting up the DVR and Tuner is also pretty simple, but the harder part (and a major reason i have 2 antennas, is actually getting them aligned.

For this I used TVFool to get a good idea of the general direction of the towers in my area.

You can put in your address and get a decent idea of the relative antenna locations

Additionally you get your relative strength and co-channel interference you may encounter.enter image description here

There is a handy color chart to help you determine the type of antenna you may need.enter image description here

For me, I was able to get by with indoor antennas so no major drilling outside. However two of my towers were about perpendicular to each other. A single large outdoor antenna likely would work better, but the lazy part of me didn't want to run a 15a plug outside for a booster. So I settled with 2 indoor ones, on perpendicular walls.

Alignment I found to be easiest by starting a stream on a channel in plex, then pulling up the HDHomerun tuners page and adjusting the antenna location until you get a suitable strength. Refreshing the page updates the stats.

The details below are about the MINIMUM needed for a solid stream or recording with no breakup.

enter image description here