Installing Kubernetes is a weekend. Running it is the other 400 nights. Most of the difference never appears in a proposal, because it consists of things that are only visible when they fail: a backup that turns out to be missing, a certificate nobody renewed, a volume that filled up at 3 a.m.
We sell the operation, not the installation. This article is what that sentence actually means in our clusters - specific enough that you can check whether your current provider does the same.
Three layers of backup, because one is a single point of failure
Block snapshots, cluster objects and logical database dumps solve different problems, and none of them substitutes for the others.
Layer one: volume backups. A recurring Longhorn job runs at 01:00 UTC and backs up every labelled volume to Hetzner object storage in Helsinki, incrementally, keeping seven days. This is a complete image of the block device - fast to restore, but only crash-consistent for a running database.
Layer two: cluster objects. Velero backs up Kubernetes manifests - deployments, stateful sets, secrets, config maps - with a one-week retention. Deliberately no volume data: that is layer one's job, and mixing the two produces backups that half-fail in ways nobody notices.
Layer three: logical dumps. For databases where we want transactionally consistent restores and "put it back to yesterday" granularity, a per-application cron job dumps the database to its own bucket, separate from the volume backups.
The default is not to back something up. Every protected volume is a deliberate line in a list in Git, and every decision not to protect something stays in that list as a commented line with a reason. Otherwise, six months later, nobody can tell whether a missing entry was a decision or an oversight.
What the monitoring covers
Alerts exist for the failure modes that actually take services down, not for a dashboard full of green:
- Storage - volumes over 80 % and over 90 % full, plus a rapid-growth alert for anything gaining 10 % in an hour. Growth caught by trend beats growth caught by outage.
- Storage health - degraded and faulted volumes, nodes whose storage is not ready, disks approaching full.
- Cluster health - control plane and worker memory, etcd database size, nodes going not-ready, I/O wait, TCP retransmits and CPU steal.
- Outside the cluster - uptime probes, TLS certificates approaching expiry, mail queue growth and disk usage on standalone servers, which report into the same Prometheus.
- Logs - centralised, so an incident is investigated from one place instead of by guessing which pod to tail.
Storage sized so it can still grow
One rule shapes how we provision, and it is worth stealing: a volume can be enlarged, never shrunk.
Kubernetes rejects a reduction in requested storage. So "let's add some headroom to be safe" is not caution - it is an irreversible decision, and a noisy one, because a later attempt to reduce the number fails the upgrade and blocks every subsequent deployment of that application. Undoing it means downtime, a backup, deleting the volume and restoring the data.
So we provision tight and grow reactively. The capacity alerts give us warning, expansion is online and needs no downtime, and enlarging is cheap while shrinking is expensive. When in doubt, the smaller number is the safer one - the exact opposite of the instinct most people have.
Production data lives on three-replica storage. Single-replica storage is reserved for things that are genuinely disposable, like caches and staging. We economise on volume size, never on redundancy for live data.
Git is the source of truth, including at 3 a.m.
Nothing in production gets changed by hand. Deployments happen through Git and the pipeline: commit, build an image tagged with the immutable commit SHA, upgrade the release from charts and values that live in the infrastructure repository.
The temptation to fix a production system directly is strongest during an incident, which is exactly when it does the most damage - the change is invisible in the audit trail and disappears at the next rollout, usually weeks later, when nobody connects the two events. Emergency access exists, but it is an explicit, recorded decision rather than a habit.
How you would leave
Since we are describing what we do at night, it is fair to describe the exit too. You can take your data out at any time: database dumps, volume contents, the Helm values and manifests that define your deployment, and the source of anything custom we wrote for you. Everything runs on standard open-source components - Kubernetes, Longhorn, Postgres, Keycloak - so there is no proprietary runtime to switch off and nothing that only works while an invoice is paid.
That is not generosity. A client who stays because leaving is painful is a client who has stopped evaluating whether we are any good.
Who this is for
- Companies with production workloads but no dedicated operations team, where "who gets paged?" currently has an uncomfortable answer.
- Regulated organisations that need to demonstrate backups, retention and recovery rather than assert them.
- Teams that already run Kubernetes and want the operational layer taken over without handing away the platform.
Getting started
Tell us what you run, what your recovery time and recovery point requirements are, and what has to keep working during the migration. We come back with a plan and a quote.
E-mail info@cybermindnet.eu or use the contact form.