INFRASTRUCTURE / SYSTEMSES ↗
Things I write down.
Technical notes about things I've built, broken or learned. No order and no schedule: I write them when there's something worth telling.
- How my homelab is put togetherTwo mini PCs running Debian, the system on the NVMe and the data on the SSD, every service in its own folder with Docker Compose and in git, monitoring with Prometheus and Uptime Kuma, and 3-2-1-0 backups to Backblaze B2. What's there, how it's organised and why.
- 3-2-1-0 backups on my homelabThree copies, two media, one off-site and zero errors: restic backups between my two mini PCs and Backblaze B2, plus a real restore every week to check they work.
- From SAN to vSAN without changing the networksA seven-host cluster with mixed versions, an old array with HDDs and one condition: don't touch the customers' networks. How we moved it to all-flash vSAN in a month, in phases, migrating the VMs live with Cross vCenter vMotion.
- Migrating customers from vSphere 7 to 8 with VeeamTwo datacenters with no direct connection between them, so no vMotion: a full Veeam backup the day before, incrementals on the day and about 20-30 minutes of downtime per customer. Storage went down from 85-90 % to 60 %.
- Keeping macOS updates from removing admin rightsEvery macOS update removed the users' local administrator permission. An Intune script, assigned to a group of users, puts it back on its own. And around it, how we retire a computer with Intune and M365.
- A VMware datacenter from scratchFive Dell hosts with EPYC 9534 and ESXi 8, Pure Storage arrays over NVMe over RoCE instead of iSCSI, and one VDS per customer so as not to drag the standard switches along. The decisions I made and what improved: restoring 1 TB went from about 20 minutes to 7-8.
- A DR with Veeam 13, actually testedVeeam CDP between a customer's infrastructure and ours: a 15-minute RPO for critical VMs, a 3-hour RTO, a full day of testing and a runbook so the customer doesn't depend on us.
- A pre-production environment that refreshes itselfEvery week, the pre-production environment of a SaaS on Azure is filled with production data: anonymised PostgreSQL and blob storage synced with AzCopy, from a bastion that can only read in production. And around it, day-to-day work on Azure, AWS and Linux.
- How this site is deployedA git push and, a few minutes later, the site is updated on my homelab. No open ports, and GitHub never gets into my network.
- A status page without exposing Uptime KumaI wanted to show my homelab's status on the site without publishing the Uptime Kuma dashboard. The solution: have the site's nginx forward just two read-only URLs.