Yeah this was always going to happen after a big rush. On any website a certain % of users that sign up won't like it and will move on. If you have a steady influx of users, you wouldn't notice it, but because of lemmys explosive growth due to reddit shitting its pants, then just like we saw a tonne of people leave at once, were now seeing a tonne of people leave at once, and now that that explosive user growth is normalising, for a short time we will see an overall decline in users until the amount of leavers normalises as well.
If were still losing people in a month, then we should be worried.