Did you try the classics: cross-validation of your data set?

Hi, 

I think that you like to define a process to label automatically the documents as relevant or not. In this case, you can define a dataset of documents with their mean scores (depending on different methods; ltc,BM25, etc.) and their major label (relevant, not relevant, not labeled) depending on all queries processed. (It's just my proposition)

For methods take a look at the link bellow.

Good luck.

http://nlp.stanford.edu/IR-book/

If you are interesting in comparing ranks then look for measures like MRR(Mean Reciprocal Rank) or rank co-relations like Kendall tau or Spearman' rho, these measure will only help you in finding the rank co-relation, if you consider one of the rank created by some state-of-the art method as baseline and then re-rank using your approach. An alternative could be that use generative method like language models to see how likely it is to generate the given query from a retrieved set of documents which will hint you towards the notion of relevance of certain documents, when you don't have relevance assessment available for your test collection. If your experiment collection already comes with the relevance assessments then the standard TREC based evaluation measures will be the preferred choice. Moreover, if relevance judgement are graded then I suggest go for Normalized Discounted Cummulative Gain (NDCG) etc. 

http://curric.dlib.vt.edu/modDev/package_modules/MidtermModuleTeam5-TRECevalFinal.pdf

http://en.wikipedia.org/wiki/Discounted_cumulative_gain

Dear Monica Ihli, just a comment regarding an analysis based on semantics, not on statistics. In my experience with graduate students from several fields of knowledge, I suggest they use  a linguistic storm (a matrix of controlled vocabulary, the document I wrote is posted in here, ReserarchGate). The results are encouraging since they - Spanish native speakers - become aware of the controlled vocabulary usage. I hope this comment may be useful for you!

Dear Monica,

Maybe this article could be of interest for you:

Mark D. Smucker, James Allan, and Ben Carterette. "A Comparison of Statistical Significance Tests for Information Retrieval Evaluation," CIKM '07, Lisboa, Portugal, 2007

You can find it on the website of Mark D. Smucker (http://www.mansci.uwaterloo.ca/~msmucker/)

Gilles

Dear Monica,

There are few available literature for Re-Ranking the query result depending on your application. Few such methods are Maximum Merginal Relevence (MMR), Intent-aware Selection (IA-Select). Rocchio’s Method can also be used with little modification. 

In case you want to compare two ranking result sets the there are also few classical and neo-classical distance metric available like Kendall'd Tau distance, Spearman's footrule, MAP, NDGC .