It's not unreadable, it doesn't exist.
Then what do you mean trained AI models are?
The ai model is trained on data and encodes unknown parts of that data in its weights.
This is data storage. Unmanageable, almost unknowable data storage, but still data storage.
If it didn't store data it couldn't learn from its training.
No? When they train AI's on data they lose control of that data. If the data is sensitive, they aren't being responsible.
GPT models are as you say dumb statistical models, I agree. But in its weights are encoded ghost images of its training data. The model being dumb is not sufficient to make the data storing itself defensible in my opinion.