All projects02 / 08
A self-hosted platform that watches, updates and rebuilds itself.
01 / Problem
A public real-estate site, a game server and several tools run on one Ubuntu VM on a Proxmox mini PC. I wanted to hear about a failure before a user did, and not have to babysit updates, so I treated the box like a small production system with a registry of what should run.
02 / Watching
A watchdog runs every five minutes and checks for containers that are down, unhealthy or restarting, a full disk, low memory, failed units and a pending reboot. It messages me on Telegram when the state changes and repeats after 12 hours, so the channel stays readable. It also pings an outside dead-man switch on every run, so if the VM, the power or the network goes down I still get told.
03 / The hard part
Once a week the updater tags the current images as a rollback point, pulls and rebuilds, and waits up to three minutes for every container to report healthy. If one does not, it restores the old images and recreates the stack. I proved the rollback on a throwaway stack first, and I keep the Minecraft server out of the weekly run because its mods have to match each game version.
04 / Result
A test run that opened browsers without a limit filled memory and swap and froze the whole VM for about an hour. Since then every heavy workload runs under a memory cap, and containers have limits too. The box can be rebuilt from one bootstrap script and a written recovery guide, and every service is its own git repo with a changelog.
How it works
Every service is a Docker Compose stack. One loop watches them, another updates them, and both end in a message to me or an automatic undo.
The same flow as a list, in order.
Stack, by layer
Live from the homelabOffline