So, I bought a nice Zotac ZBOX with 2TB local storage and 16 GB memory. Installed Debian with k3s and partitioned the disc to have a ZFS pool for the containers. I've set up the ZFS provisioner.
I'm migrating away from an RPi4 with 4GB of memory and docker-compose
I run out of memory and have my pods evicted while not having all my things migrated. Okay, I've got two new things: ArgoCD and Navidrome. But what the heck?
Now I've updated k3s according to the manual (just download the binary and restart). It wouldn't start. Ran the installer instead. During boot-up, two of my ZFS volumes got deleted by the provisioner. F*ck. Luckily no important data on there, just extra work to redo the migration. That's the second time the provisioner kissed a volume goodbye. The first time, I was to blame because I messed up a commit in my ArgoCD repo. Data was faster gone than I could say bye-bye.
I work with Kubernetes daily. K3s seemed a logical choice for managing a more significant environment. I looked into Proxmox, but docker containers are not supported.
I spent every free minute for the past 14 days migrating, but I have my doubts about the correct path. Eventually, I need a stable system.
ZFS was supposed to save me from some of those stupid mistakes. I run Btrfs so far, and snapshots are a lifesaver. Can't wrap my head around them in ZFS. Unfortunately, is the Btrfs provisioner not very widespread/stable.
Any advice or encouragement?
PS: Velero is still on the todo list... :(
Set the max ark size to something like 2GB if that's what's eating the RAM
This. You need to tune the zfs memory, esp if the box is shared with other applications.