Hi, Nadeem!

Let's dive in deep how the most of clustering algorithms work. This allows us

to find what kind of encoding should be used to handle such categorical data.

If we consider hierarchical clustering algorithms, they use distances between

objects/items to be clustered and form groups according some linkage rules.

So, distances between objects are very important when do clustering.

See full version of the answer here: https://gist.github.com/scidam/2b5dfb75556c33124f9a089f66660396

Dear Dmitry 

Indeed a very  good explanation and answer I must appreciate it. Now let me unfold my question more. This is the topic I am researching since sometime.  as mentioned earlier I have 6 columns , the other columns does not have cyclic information for example  the third column are the names of  subject categories such as English, math, science etc. and rest of columns are of the same nature. The initial approach I used was to represent all of these column with vector space model and then calculated cosine distance function between each of these column and later used this cosine  distance function in k-mean algorithm. However  while digging into python I found most of clustering algorithms need numerical representation of the data so Enable Encoder was my first choice. Let me know what are your thought of using vectorization techniques  mentioned earlier in the message. I would certainly check one hot encoding  and would than examine the accuracy of the cluster. thank you.

Dear Nadeem,

It is better to binarize categorical data before clustering. Binarizing yields good distance.

I think one of the best choices for encoding categorical data is OneHotEncoder; cosine distance will be also suitable; cosine-dist is less dependent on absolute values of vector's components: e.g. if we consider vectors (10, 1, 1) and (50, 1, 1), where the first component is of floating point nature, but the rest are  results of binarizing, the cosine dist will be 0.00629..:

import numpy as np

>>> x = np.array([10, 1 ,1])

>>> y = np.array([50, 1, 1])

>>> cos_xy =1 - np.dot(x,y) / np.linalg.norm(x) / np.linalg.norm(y)

>>> cos_xy

0.0062892716776725699