I suspect it goes deeper than language even. We have associative memories, right? Given our learning, modelling, pigeonholing of the world along with our changing bodies, our older selves are unlikely to experience anything close enough to what our younger selves experienced to trigger that association and recall the memory.