I want to setup a search engine performance system to automatic estimation.
Say we have a dataset with more than 30,000 documents, its hard to find relevance data for some queries.
which measure is better for this task?
MAP, nDCG, bpref?
Some says bpref is better, but paper[1] says it have average performance. Can somebody help what is a better measure which can be setup with less efforts.
[1] https://pdfs.semanticscholar.org/c02b/3ec648307a759604c4374b162e9c02a58667.pdf
Personally, I think you need to evaluate the system from different perspectives (such as its ranking ablility and its retrieval ablility). Hence, I recommend you involved at least two different measurements for example F1-score for its retrieval performance and nDCG for its ranking performance.
Jiayin Lin thanks, can you please share some relevant links?
Hi, here is the link about measurements: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)
But, I highly suggest you have a look at relevant papers in this area (top conferences or journals), and see how the evalutate their models.
Thanks Jiayin Lin , yes I have few papers regarding these.
Currently I am also setting up a method to get user feedback on my search results rankings
the F-measure is also a performance measure to evaluate the IRS. it combines the recall and precision measures.
- Badges
- Science topic