Related to your query, I suggest you follow Article Construction and evaluation of gold standards for patent cla...

https://www.aclweb.org/anthology/P99-1032.pdf

Actually I want to use the classification golden standard to "conclude" relevance judgments for text retrieval

Nina Saabiyeh ,

Are the documents annotated by sections? If they are I think it is doable.

Regards

I don't know what you mean with "annotated by sections", but the paper which the dataset was originally created for say that experts annotated the papers by reading (title, abstract and keywords) of each paper, and then selecting annotion topics.

Nina Saabiyeh ,

What I meant with annotation by section is that each section of the document is treated independently and annotated independently. for example a book on developmental psychology might be divided into new born section, infant section,..., etc and annotated. these sections might then be treated as different documents.

But from your answer it seems that it is not.

What you have is a complete document with just 1-N classification. In that case the best that can e done is to try to equate documents of a single classification as the highest and documents having multiple classifications as the lowest. But these less granular the segregation the less reliable it becomes. If for example, you had a document with three classifications and each paragraph tagge with the particular classification, then it might provide a better outcome.

Regards