I could be misinterpreting your claim here, but I'll point out that LLM weights don't literally encode the entirety of the training data set.