That’s been my observation too. My completely amateur theory is that a young child’s perception of the world - their ability to contextualise what they’re looking at - changes so much that even if those old memories are still hanging around, they don’t really make sense any more. It’s more like a sliding window than a hard cutoff.
I read a similar discussion here on HN before and the explanation that its how our brain decodes memories shifting over time really hit me. I have a theory that if that in fact is the case, then things like journaling would help to retain memories in a superior way, since we can review such things as our encoding changes and retain it while we can still decode. Even if we dont remember it as richly though we will still be able to go back to a journal to remember something at a bare minimum.