Fleiss kappa can be used when you have nominal variables while ICC can be used when you have ordinal, interval, and ratio variables.If the ratings use a Likert scale or a numerical value then the ICC might be a better way to go.

You could look at this paper:

Dear Mr. Ali Ghaffari:

I will read the paper you recommended. Do you know any textwise literature on this issue? If any, please let me know. Thank you for your help.

A major textbook that covers this issue is Krippendorff, Content Analysis: An Introduction to Its Methodology ...

Hi.

Young-Jun Choi.

Perhaps you want also consider the application of even swaps (https://hbr.org/1998/03/even-swaps-a-rational-method-for-making-trade-offs).

Rgds.,

Klèber

Dear Professor David L. Morgan:

Thank you for your advice. I borrowed the Krippendorff's book (2nd edition) yesterday and will read the relevant section.

Dear Mr. Kleber Cervantes:

The article you recommend to me seems very interesting. I am not sure that even swaps will be used in my ongoing work, but it is worthwhile to review carefully. Thank you for your comment. 

I, personally, found this short text book one of the most comprehensive & informative:

Kilem L. Gwet; "Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters; Fourth Edition"; Copyright 2014: Advanced Analytics, LLC.

Dear Dr. Anthony Waddimba:

Thank you for your suggestion. After reviewing its table of contents, I thought that the book you had recommended might be one of the most comprehensive books on this topic. I will ask a librarian at my institution to buy its recent edition.

Thank you again.

The weighted average method also can be used. The weight of each raters can be assign by different criteria such as, experience, frequency of website use, and others.

Thank you for your advice, Professor Feng. I will read your paper.