Recently when I tried to visualize similarities between different groups in a social network, I discovered an interesting phenomenon which I couldn't explain. Can someone give me some insight on this?
The story is as follows: Users can create different groups on our social network website, and we'd like to visualize the similarities between different groups in 2D. We use the group members and their page views within the group as the feature vector of each group, and the similarity between groups is computed as the cosine similarity between their feature vectors. I used multidimensional scaling ( cmdscale in R ) to reduce the data into 2D and visualized the data.
The result of the MDS is points lined up with some lines orthogonal to each other. Can someone explain why this is happening?
The figure looks interesting, but your question is unclear. I don't understand what a point in your figure represent for. What are the x axis and y axis?
cmdscale, if i remembered correctly, need the input to be a distance/dissimilarity matrix. Did you transfer your data? Did you try to calculate the stress value to see the distortion of your 2d map?
I thing, the axes don't have any interpretation...this is just 2D field to place the figure. This figure could be rotated in any direction. The only thing that matters is the relative position of each point and its distance to the neighbours.
Could you try to account only the common member within groups? (I mean without information about page views)....
Before I try to make comments, I want to state my understanding of the problem.
• I assume that we are going to make many observations, each one spanning a week or so, and that the number of added pages between observations is negligible compared to the total number of pages (N) in the in your social network website;
• I assume that each group is going to be represented by just one featured vector of N components. The value of each component is the number of views that altogether the members of the group have accessed the respective page during the observation period (or you can normalize to max=1);
• Then if you have k groups, you going to have 0.5*k*(k-1) cosines to observe similarities or dissimilarities.
• If k is large then MDS is a good tool, but you have to check the stress function to get a feeling of what how well MDS is helping you;
• In your case, the surprising results, could be (wild gess) that you were not representing the vectors with the same basis, (may you are just considering as dimensions the pages generated by the group) then MDS will probably reduce to something like your picture.
• If, under proper representation, MDS is not helping try clustering, PCA + clustering, etc
As per my understanding the two dimensions mean that the members of the social group may belong to two diverse interest groups. However there is a person in the group who has been equally interested on both the dimensions, So possibly, that person is a pivotal element of the social group and possibly the most important person in that group. For instance, suppose it is a group of people with financial and marketing background. Somebody with lets say having background of production/R&D may be interested in both financial and marketing aspects may become the pivotal or orthogonal element.
What are the "features" quantified and how. I think we need more information to make sense of the plot.
Whatever value, a, of the attribute on one abscissa always gives a constant, f(a)=const, on the other. I'd say that the attributes' sets are distributed independently of each other.
I guess some coordinates on the feature vector are linear dependent on an important subset of the visualized data, so projection on 2D space are necessarily reflected as straight line.
Thank you, guys. Let me explain more clearly what the feature vectors are. Each group has only one feature vector. Let's say all the users in all groups are labeled by {0, 1, 2, 3}, group A has members 0, 2, 3 with page views 45, 75, 65, therefore the feature vector of group A would be [45, 0, 75, 65]. There is a 0 in position 1 because user 1 does not belong to group A, hence its page view is 0.
@Hao, It seems interesting! Since you have the similarity matrix of groups, have you applied other methods to see if there exist 4 overlapping communities? I am just curious. thanks.
I agree with Sajid. (Though it could be an artifact, too.)
The reasoning is: each point is a group - originally in num-users dimensional space. You derive distances using cosine similarity. Then you use classical multidimensional scaling to reproduce these distances approximately in 2D space.
In this space the distances between certain pairs of groups are very small. This closeness is transitive, leading to the lines. Thus, one can interpret the lines as communities - people who read one group are likely to have overlapping interests in a nearby group.
The groups nearer to the center have broader fan-bases, they are relatively close to more than just two neighbors. This indicates significant overlap - though not as much as the overlap between two groups near one another in the same line, but still more than the overlap groups at opposite ends of the same line would have.
The orthogonality comes, in part, from almost complete lack of overlap between groups on the tips of each line. Those would contain the most specialized interests of each community, which only members of the same community are interested in. I would look at those groups to find clues to simple descriptions of the communities.
I used VOSviewer to visualize the data. The result is in the attachment. There, too, are points lined up along a straight line. I think this is because in my case, relatively few groups are similar to each other, hence they form a line. If there are many groups that are similar to a single group to the same extent, there would be circles or cliques.
I sent a message to the author of VOSViewer, asking the same question. He explained that because the groups are very distinct from each other, the visualization degenerates to one dimension. I checked the similarity matrix. Indeed, groups are very distinct. Today, from the book "Mining of Massive Datasets", I learned any two pairs of vectors in high dimensions are almost always orthogonal to each other. This explains why my groups are so distinct from each other - the dimension is around 150000. Since VOSviewer and MDS can not process too many small similarity values, I think this implies that they do not work well for high dimensional data.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining