Linux

Foreground targets are where writes initially go. Data is moved from foreground to background targets while idle or as needed. Data which is read from the background targets is moved to promote targets.

If you set your NVMe as a promote target, SSD as foreground and your HDDs as background targets, all writes would first go to your SSD, then get copied to your HDD during idle, and finally the copy of the data on your SSD will then be marked as a cached copy. In case your SSD becomes full, then it'll store the data on other drives. As for the promote targets, any time you read data from either the SSD or HDD that wasn't on the NVMe, it would get cached to it, so the next read will be faster.

The main point of the foreground vs promote is to prioritize write vs read speeds. If you value faster writes, then set your NVMe as foreground. If you value faster reads, then set your NVMe as promote. Of course, you can also set your NVMe as both foreground and promote to benefit from both faster reads and writes.

But since you plan to introduce an SSD in the mix, you can create a single group for your NVMe + SSD, and a second group for the HDDs, and set your SSD group to foreground + promote, which will simplify things.

The Arch wiki illustrates this well:

A recommended configuration is to use an ssd group for the foreground and promote, and an hdd group for the background (a writeback cache).

Modified example to your scenario:

# bcachefs format \
    --label=ssd.nvme1 /dev/nvme0n1 \
    --label=ssd.ssd1 /dev/sda \
    --label=hdd.hdd1 /dev/sdb \
    --label=hdd.hdd2 /dev/sdc \
    --replicas=2 \
    --foreground_target=ssd \
    --promote_target=ssd \
    --background_target=hdd 

If you're concerned about chucking both the SSD and NVMe in the same group, no need to worry cause bcachefs will automatically prioritize reads from drives with lower latency as mentioned in the wiki.

If they are different speeds, reads for replicated data will be sent to the ones with the lowest IO latency.

But regardless of which setup you go for, main thing to remember is to use the NVMe (or the group containing the NVMe) as the promote target, as that will be your primary cache drive.