Dear Qiang,

Your question is very relevant to assessing expert opinions. Let me assume that reliable means independently replicable by others, rather than consensual. I have used methods from voting and ranking. The methods are axiomatic an thus can be said to be sound. The simplifying assumption is that a single public decision is accountable and responsible to society. Most, if not all, regulatory science-policy contexts rely on integrating judgments, use direct and indirect evidence (i.e., the probative evidence just discussed) for their analyses, and select aspects of each of the following sets::

{literature adequacy: e.g., summarizing automated findings of key characteristics of causal system}, {adequacy of the research used), {assumptions), {modeling forms}, and {results}.

The integration includes: i) individual expert assessments and ii) formal rules for integration, such as majority voting, on specific scientific propositions such as models and results. Regarding the rules, collective objectives include procedural fairness and epistemic truth (a limited form of truth. For example, the rules may be:

1) Proposition-wise voting – ranging from plurality to absolute majority,

2) Proposition-wise quota – threshold based, may include a distance metric between the conclusion and the centroid of clusters of individual experts’ choices, or

3) Conclusion-based voting – ranging from plurality to absolute majority.

Ordinal ranking of preference over choices is another example of a collective assessment. Look at Arrow’s five preference axioms over choices. These are: i) universal domain; ii) ordering (i.e., completeness and transitivity); iii) independence of irrelevant alternatives; i) weak Pareto ordering; and v) non-unique and dominating alternative. Briefly, completeness prevents asymmetric information by requiring all possible profiles to be accounted. Ordering implies linear ranking; irrelevant alternatives should not affect rational ranking. For Pareto ordering, if pairwise ranking yields a choice and that choice is similarly ranked across all pair-wise comparisons, then their aggregate must reflect this.

See List (2012) for the preference relationships for these axioms; Dietrich and List 2007b; Dietrich and Mongin, 2010 for more details. Some critical references are:

List, C, The theory of judgment aggregation: an introductory review. Synthese, 187 (1). pp. 179-207. ISSN 0039-7857 DOI: 10.1007/s11229-011-0025-3 (2012).

Dietrich, F and List C, 2004, “A Model of Jury Decisions Where All Jurors Have the Same Evidence.” Synthese, 142: 175–202.

· –––, 2007b, “Arrow's theorem in judgment aggregation.” Social Choice and Welfare, 29: 19–33.

· –––, 2007c, “Strategy-proof judgment aggregation.” Economics and Philosophy, 23: 269–300.

· –––, 2007d, “Opinion pooling on general agendas.” [Dietrich and List 2007d available online (pdf)]

· Dietrich, F. and P. Mongin, 2010, The premise-based approach to judgment aggregation. Journal of Economic Theory, 145: 562–582.

L.S. Shapley, B. Grofman, "Optimizing group judgmental accuracy in the presence of interdependencies" Public Choice, 43 (1984) pp. 329–343.

Hope this helps!

Paolo

a nice question

Dear Qiang,

Your question is very relevant to assessing expert opinions. Let me assume that reliable means independently replicable by others, rather than consensual. I have used methods from voting and ranking. The methods are axiomatic an thus can be said to be sound. The simplifying assumption is that a single public decision is accountable and responsible to society. Most, if not all, regulatory science-policy contexts rely on integrating judgments, use direct and indirect evidence (i.e., the probative evidence just discussed) for their analyses, and select aspects of each of the following sets::

{literature adequacy: e.g., summarizing automated findings of key characteristics of causal system}, {adequacy of the research used), {assumptions), {modeling forms}, and {results}.

The integration includes: i) individual expert assessments and ii) formal rules for integration, such as majority voting, on specific scientific propositions such as models and results. Regarding the rules, collective objectives include procedural fairness and epistemic truth (a limited form of truth. For example, the rules may be:

1) Proposition-wise voting – ranging from plurality to absolute majority,

2) Proposition-wise quota – threshold based, may include a distance metric between the conclusion and the centroid of clusters of individual experts’ choices, or

3) Conclusion-based voting – ranging from plurality to absolute majority.

Ordinal ranking of preference over choices is another example of a collective assessment. Look at Arrow’s five preference axioms over choices. These are: i) universal domain; ii) ordering (i.e., completeness and transitivity); iii) independence of irrelevant alternatives; i) weak Pareto ordering; and v) non-unique and dominating alternative. Briefly, completeness prevents asymmetric information by requiring all possible profiles to be accounted. Ordering implies linear ranking; irrelevant alternatives should not affect rational ranking. For Pareto ordering, if pairwise ranking yields a choice and that choice is similarly ranked across all pair-wise comparisons, then their aggregate must reflect this.

See List (2012) for the preference relationships for these axioms; Dietrich and List 2007b; Dietrich and Mongin, 2010 for more details. Some critical references are:

List, C, The theory of judgment aggregation: an introductory review. Synthese, 187 (1). pp. 179-207. ISSN 0039-7857 DOI: 10.1007/s11229-011-0025-3 (2012).

Dietrich, F and List C, 2004, “A Model of Jury Decisions Where All Jurors Have the Same Evidence.” Synthese, 142: 175–202.

· –––, 2007b, “Arrow's theorem in judgment aggregation.” Social Choice and Welfare, 29: 19–33.

· –––, 2007c, “Strategy-proof judgment aggregation.” Economics and Philosophy, 23: 269–300.

· –––, 2007d, “Opinion pooling on general agendas.” [Dietrich and List 2007d available online (pdf)]

· Dietrich, F. and P. Mongin, 2010, The premise-based approach to judgment aggregation. Journal of Economic Theory, 145: 562–582.

L.S. Shapley, B. Grofman, "Optimizing group judgmental accuracy in the presence of interdependencies" Public Choice, 43 (1984) pp. 329–343.

Hope this helps!

Paolo

To add to Paolo F Ricci 's excellent overview, I would note that "decisions" usually are made with limited data. Human "Risk Assessment" will vary between circumstances and individuals. Thus, identical data may result in different "decisions" either at different times or from different individuals (or groups). Our own feedback mechanisms are to see if the outcomes are really what we intended or expected. Too often in societal guidelines, they are not, yet they often go uncorrected or revoked.

We select (and retain) leaders of business and society (in most cases) based on their subjective performance in risk management of all kinds. Cultural risk aversion or acceptance constantly change as well, making a quantitative projection of decision-making "reliability" subjective as well, and nearly impossible across all times and circumstances.

One "danger" of things like "Artificial Intelligence" is a propensity of people to incorrectly assume that all relevant data can be made available, and that such availability will lead to inerrant solutions of problems.

Best regards,

Steven

Dear Paolo F Ricci , thanks for your excellent answer.

Dear Steven Cooke , I couldn't agree with your more. However, I want to know if there are any methods could decrease the subjectivity of experts' evaluation information.

Qiang Yang , YES, there are mechanisms to mitigate subjectivity in hazard and risk assessments! Most common are agreed on matrices of an incident probability and consequence that at least rank relative risk consistently within a particular organization. Some of the metrics are more widely accepted, others may vary more (or simply remain internal to a company). Others (Kano Model) use a more 'fuzzy' impression to find preferences, but I find that less useful for risk issues. I can provide you with a detailed example if you post your email address.

Dear professors,

Prof. Paolo's comments and suggestions are very nice. They seem to come from some angles of social science and decision making experience. Just, I think Prof. Qiang Yang need some numerical model (which might be as simple but effective as possible) to solve some certain problem he faced or pondered upon.

The question at this stage seems not easy, and deserves further study.

Maybe, one can consider the uncertain nature of those opinions provided by experts in decision making. In MCDM, the individual evaluation those experts provided can have different types of uncertainties, e.g., with fuzziness, probability, interval or basic uncertain information (BUI, as we recently proposed).

Then, some data aggregation and information fusion method (like OWA) can be used to obtain a comprehensive evaluation/result by weighting more on those individual evaluations with higher certainties/lower uncertainties, and weighting lightly on those with lower certainties/higher uncertainties.

This is a simple (but effective) numerical method to “decrease the subjectivity of experts' evaluation information” from some special perspective.

Thanks.

Lesheng Jin Thanks for your answer. Additionally, could you give me some suggestions on how to determine the uncertainty of experts' evaluation information in MCDM.