I have calculated similarity scores from 302 textual stories. How can I use more representative similarity scores? I am afraid that similarity matrix tells us the similarity score of each story with the rest of stories in a column.
What if I want to find the overall similarity measures for each story in my corpus?
Lets suppose we have 3 stories
A is 0.53 similar to B
A is 0.73 similar to C
I want to find out the overall similarity of all A, B and C like below
A = 0.69
B = 0.53
C = 0.44
In this way, I would be able to use it in SPSS or ML to find important correlations or predict something.
Note: Data is fictious
Dear Tahir,
this is a very trick question. I'm seeing your problem as one another that I faced some time ago. The approach that I used was to perform a "against-all" comparison.
Imagine that we have four stores (a,b,c,d)... I made the following comparisons: a-a (equal to 1, for sure!), a-b, a-c, a-d, b-b (1 again), b-c .... till d-d. At the end... I obtained a triangular matrix with overall comparisons.
As a index of the story a I used the median of the line 0, for b... the median of line 1 and so on. Off course that you can evaluate statistic measure to your problem.
I'm very interested in your results. Please, add me and let me know if this approach was suitable for you.
Regards,
Fábio
Dear Fabio,
Thanks a lot!
I am wondering why not you used cosine similarity! It does exactly the same. However, I am interested in the second part of your story: the median.
I used mean. What was your reason to use median? could you please explain?
Have you published a paper based on those results? If yes, I would appreciate if you can send me the reference as well.
Thanks for your time.
Hi Tahir,
I used median because I had few data and I was expecting to observe outliers. In case you don't have outliers for sure, use the mean.
I didn't publish these results because it was developed for a company and the confidentiality agreement is now allowing any publication for now.
Thanks for your reply!
Regards,
Fábio
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python