In semantic similarity a benchmark data set is used to measure the effectiveness of semantic similarity measures. SemEval-2017 and earlier workshops uses the Pearson correlation to judge the effectiveness of proposed semantic similarity methods. One major problem for Pearson correlation is the outliers that influence the slope of the regression line, and consequently on the value of the correlation coefficient. Moreover, The Pearson correlation does not have the ability to show wither a similarity method is good for non-similar, average-similar or similar text fragments.
Is it time to find an alternative way?
if yes, what do you suggest?
Hi,
Converting this problem into a regression modelling task can be used to handle this issue and answer your question though. The misclassified instances are considered as outliers in this case. Moreover, the correlation coefficient can be used to assess the quality of the resulting model.
NB: All the texts have to be converted into a word vector.
HTH.
Samer, PhD.
The issue is the output of unrealistic coefficient as Pearson correlation is dependent on mean values of similarity scores. Therefore, in many cases the comparison of similarity methods is not ground.
When there is a label for each instance, you'll have a supervised learning problem. Methods like cross-validation evaluates this learning efficiently. Correlation coefficient, in this case, is considered to be reliable.
HTH.
Samer
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs