One of the problems with doing your own thing with maintaining your own servers, hosting, blogging or other website pursuits is you break things when you experiment and that happened with this site. Pretty ugly really considering this site became a complete loss.
This site is hosted within a Kubernetes cluster and consists of two pods. The blog itself and the backend database for the pods. For each of those pods a persistent volume is assigned for the pod data theoretically in order for that data to stay persisted after dropping the two pods.
Well… that’s the theory anyway. One day I discovered my cluster was acting wonky. Since the cluster itself is managed by Digital Ocean I fired off a few emails to support to figure out what was happening. Long story short I had overloaded my workers and needed to resize them. Did that and everything looked like it came back to normal.
I have several blogs on this cluster and I checked them and they were back to normal. I didn’t check this one and since it looked from the command line like the pods were all happy I left well enough alone. Big mistake.
What I didn’t know was the WordPress application was slowly tearing itself apart. I am not sure about the mechanism but it was certainly tearing itself apart. Slowly but surely the database became corrupted and the data pv for the blog became corrupted and today the day before April Fool’s day 2020 I found it totally insane. So were the backups corrupted.
So I’ve started over again. For about the fourth time. No more experiments on my “production” blogs any more.