Harnessing ZFS Filesystem for Your Homelab: Tips, Caveats, and a Reference Goldmine

Here is a list of known ZFS downsides that didn't stop me from using ZFS on my homelab.

• It's memory-hungry. From what I've read it's not recommended to have less than 8GB of RAM with ZFS. In my case, it consumes about 6GB of RAM under load, like making a time machine backup.
• There are limitations to how you can extend your ZFS pool. Long story short: ZFS is not very suitable for growing the storage gradually. If you have some kind of an established ZFS RAID, you cannot simply buy a new drive and increase capacity. Currently, folks are working on a solution to make ZFS pool expendable.There is a pull request which is not been merged yet.
• Native ZFS encryption is slow. It's slower than LUKS with ZFS on top of that.
• There is an opinion that ZFS is just overcomplicated and not mature enough and it's more reliable to just use EXT4. For example, there are issues with encrypted storage replication resulting in corrupted snapshots.
• ZFS is not supported natively on MacOS. There is an OpenZFS package. It works. But it doesn't support all the features.

ZFS Explained

• The best two articles on the internet explaining what ZFS is: "ZFS 101—Understanding ZFS storage and performance" and "What is ZFS? Why are People Crazy About it?"
• Here is a worthy overview of ZFS on Proxmox wiki

Configuration

• The author of the thread has a number of disks of different types and sizes and is questioning about storage configuration options. The thread has worthy advice.
• The ZFS tuning cheat sheet page contains the ZFS configuration parameters explanation as well as the optimal values of choice.
• The post describes a useful example of ZFS pool naming and dataset hierarchy organization for replication and backup needs with thoughtful data prioritization.

Snapshots, Backups and Replication

• Docs with all needed info about ZFS replication: Sending and Receiving ZFS Data
• Here is a worthy article describing ZFS replication in detail: How to use snapshots, clones, and replication in ZFS on Linux
• Folks are discussing caveats of replicating native ZFS encrypted pool here and here and there

Encryption

• ZFS native encryption kick-off article
• OpenZFS on OS X encryption support
• Tutorial on how to backup encrypted ZFS data on untrusted storage without the need of unlocking it.
• "Secure External Backup with ZFS Native Encryption" - detailed tutorial of setting up ZFS backup
• Discussion of ZFS Encryption options on reddit
• Proxmox failover cluster with encrypted root and zfs encrypted storage with auto-unlock based on Debian 11 (LUKL + ZFS)
• How to install LUKS encrypted Server and enable remote unlocking tutorial

Comparison of different encryption options. EXT4 vs ZFS native encryption vs LUKS vs BTRFS and different combinations comparison

Key takeaways:

• ZFS native encryption is slow.
• LUKS works nicely with any of BTRFS, ZFS, EXT4

Cloud Storage For Offsite Backup

There are caveats of using ZFS with offsite backup storage.

All magic of incremental backup features of ZFS are working on a filesystem level. It means that if you want to make incremental backups to an offsite storage that storage should also be ZFS.

However, if it's not ZFS, you can still back up with something like rclone just like with a usual filesystem, but it makes using ZFS a bit purposeless.

Luckily there are ZFS cloud storages.

• ZFS Rent. Send these guys a hard drive and they will set up a storage instance for you.
• ZnapZend
• Rsync.net

One of the best non-ZFS 3rd party cloud storage for backups I've found is probably backblaze.

There are options allowing to send and store raw ZFS snapshots on any storage. However, I've got concerns about this idea.

I'm not a fan of that idea though. Native ZFS tooling performs checks allowing to ensure that snapshots are not corrupted. This is important because snapshots are incremental. I'm concerned that the whole data will be lost if any of the raw snapshot gets corrupted for some reason.

Using ZFS with Hardware Raid Controller

Discussion on using ZFS filesystem with hardware raid

Key takeaways:

• ZFS's software RAID works best when it's aware of hardware configuration.
• Hardware raid hides underlying hardware config which makes the ZFS redundancy feature less efficient