In my experience, pre-trained models are not suitable for unsupervised tasks. Especially in deep clustering when pre-trained models are used, they often have worse results than without pre-trained models.
What is the scientific reason for this?
Why are the learned representations in pre-trained models not suitable for clustering?