In general you do not have to distinguish latent variables with parameters, by altering their position w.r.t Bayesian procedure one can yield different pictures from a single model. For example, you can either cancel out latent variable or weight matrix from a linear Gaussian model, which results in probabilistic principal component analysis(PPCA) and Gaussian process latent variable model(GPLVM) respectively.

If you are only concerned about K-means cluster and Gauss mixture model(GMM), then I suggest that GMM is a Bayesian version of K-means. By "Bayesian" I mean GMM does not seek an local optimal point-estimation for the latent variables. It figures out a distribution of latent variables and integrates them out. Therefore latent variable can yield a more robust model since they can refute point-estimation in this context.

But there are other ways to cope with uncertainty(missing data, ambiguity, etc) apart from probabilistic methods. You can refer to a class of "fuzzy" algorithms. Which is late in fashion, e.g.fuzzy cluster algorithm(FCM).

Hope it will be helpful.