Thank you for that offer. I am familiar with pairwise comparison methods (including AHP) and have even developed some new ones (see my 2006 EJOR article). What I am asking here is not about specific methods for eliciting judgment (although I am always interested in learning new ones), I was seeking to find if there was an underlying theory about expert judgment. The closest I have found is Magne Jorgensen's 2006 working paper (linked below). Any insights appreciated. Thanks.

http://simula.no/research/se/publications/Jorgensen.2006.8/simula_pdf_file

Thank you for that perspective. I recently came across a vein of literature about evidence-based software engineering (link below) and the notion you provided hadn't occurred. At least from my preliminary review of EBSE which involves expert judgment in effort estimation, the evidence-based work does not seem to try and supplant the expert judgment, rather it seems that the expert judgment is considered evidence. In cases where there is no evidence or data (as is often the case in risk management, and less so but also in project management), we must rely on expert judgment.

http://community.dur.ac.uk/ebse/

I look forward to reading your work. Thanks!

Yes. And I'm a fan of that legal perspective of evidence as illustrated in Crawford-Brown's continuum of evidence (below). Expert judgment is experiential insight which is definitely valid under the proper circumstances and when properly elicited.

Direct Empirical Evidence: Direct measurement of the phenomenon of interest under the condition of interest.

Semi-Empirical Evidence: Indirectly obtained from the domain of interest when the context differs in some significant way. Often, in this case, results from one study or analysis must be extrapolated to another context.

Empirical Correlations: Obtained from the domain of interest, but captured using an associated effect rather than the exact/desired effect (often, through inference).

Theory-based Inference: Using well-developed theory, risk can be calculated based on observations of proxy or intermediary phenomena.

Existential Insight: Qualitative or quantitative judgments based upon expert judgment formed through relevant training and experience.

Crawford-Brown, D. (2001). Scientific models of human health risk analysis in legal and policy decisions. Human Health Risk Analysis, 64(4), pp. 63-81.

Clifford: I appreciate your comments on the gap between "theory" and practice when it comes to eliciting and documenting expert judgment. Unfortunately, expert judgment is often treated much too imprecisely and what passes for expert judgment is not.

Would you be willing to provide me the English law that prescribes the "chain of reasoning." I'd be interested to explore what elicitation process the legal system is requiring. Are you familiar with the following article? I have just ordering it and will review it once I receive it.

Schum, D. A. (2009). A science of evidence: contributions from law and probability. Law, Probability and Risk, 8(3), 197-231.

In other fields, it is also expected that experts judgments and the rationale for those judgments be recorded and available for public review (although the information may be de-identified). For example, see the following link:

http://www.iaea.org/inis/collection/NCLCollectionStore/_Public/31/016/31016470.pdf

Scrutability or accountability is one of the principles of "rational consensus" described by Cooke.

Cooke, R. M. (1991). Experts in uncertainty: opinion and subjective probability in science.

While these principles do not usually get fully implemented with any regularity, the closer we can get to achieving rational consensus, the more "scientific" our expert judgment will become.

Thanks!

This is all very interesting and I will continue my exploration in this area.

With just a quick reading at this point, the idea of cross-examination of the expert witness sounds intriguing in my context. To some degree, we examine precision and accuracy via seed variables and how well experts' distributions contain actual values and are scored accordingly. However, I'm not sure how much the logic and consistency are probed. I'll have to think about whether that meakes sense in our context.

Check out the occurrence of "expert judgment" versus "expert opinion" using Google books Ngram Viewer: http://bit.ly/M5wxlH

Could I suggest you have a look at the work of Sir Geoffrey Vickers on 'appreciation' and 'Appreciative Systems'? (The Art of Judgment, 1965, is a good starting point.) I used his work as a way of thinking how 'experts' gain, apply and reflect upon their expertise (e.g. see West, D. (1990) Appreciation','expertise', and knowledge elicitation: the relevance of Vickers' ideas to the design of expert systems, Journal of Applied Systems Analysis, Vol. 17, pp71-78.

Daune:

Thanks for the leads. I just reviewed your 1990 and 2011 articles and ordered Vickers' book. This is a new perspective for me to consider and I'll have to digest it a bit.

I like the notion of appreciation as a vehicle for developing expertise. It reminds me a bit of the pattern recognition definition that Kahneman uses to describe expertise.

I have a couple of questions (pardon my ignorance in your domain):

1. In KE/KA, it seemed that you described several taxonomies for differentiating types of knowledge. Is there commonly accepted taxonomy of knowledge/expertise? It seemed that KADS was decomposed into four layers, but it wasn't clear to me if these represented levels of complexity or something else (e.g. mental processing, etc.).

2. It seems like the expert knowledge that you are trying to elicit using AIM for expert systems is much broader than the type of expert judgment we are trying to elicit in project management (e.g. project activity duration, cost, risk). How do you think the AIM process could be applied in project management?

Thanks again. --Paul

Hi Cliff:

I am somewhat familiar with the work on intuition (including Hogarth's). I will check out the chapter you have suggested as I have not yet read it.

Have you read Kahneman's Thinking Fast and Slow?

http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555

http://www.youtube.com/watch?v=mWaIE6u3wvw

Its a great summation of his work and goes into a fair amount of detail about intuition, what he calls fast thinking. I think you might find it interesting in your context given that intuition can often fail us.

Enjoy.

--Paul

Hi Paul,

Good questions - I'll try to answer:

1. In KE/KA, it seemed that you described several taxonomies for differentiating types of knowledge. Is there commonly accepted taxonomy of knowledge/expertise? It seemed that KADS was decomposed into four layers, but it wasn't clear to me if these represented levels of complexity or something else (e.g. mental processing, etc.).

The more I looked into how researchers 'classified' expertise, the less it made sense. Decomposing complexity into more layers did not seem to be to be particularly useful since the problems remained. For example, we have things we know (the rule-based stuff) and then we have the intuition, the hunches, the internal, personal sense --making processes that somehow, seem to emerge out of our 'being' expert. It was these latter aspects of expertise that I was interested in looking at - how could I help experts consider these attrributes and make them explicit? So, I put reductionism to one side and started looking at holistic explanations. This felt far more satisfying and Vickers' work pulled it all together for me. The soft systems work of Checkland is claimed to orchestrate Vickers'

work on appreciation so this gave me ideas about how I could apply VIckers' work in practice.

Second question:

2. It seems like the expert knowledge that you are trying to elicit using AIM for expert systems is much broader than the type of expert judgment we are trying to elicit in project management (e.g. project activity duration, cost, risk). How do you think the AIM process could be applied in project management?

I'm not sure I'd say it was 'broader' but certainly more hidden, tacit. So for example, a fairly recent use of AIM was to see if we could use it to elicit someone's 'feelings' - both physical and emotional. The feelings of the expert I worked with were absolutely fundamental to his expertise.

AIM is based around the idea of asking questions about 'purpose' (intention) and the explanation or justification of the purpose (very subjective). Its been used in all sorts of settings and domains: mental health, business planning and strategy, management training, website development, requirements engineering, sport.

If you want to give me a scenario I'll see if I can illustrate how it could be used in PM.

Daune:

Thanks for the response. As I reflect, I am more familiar with AI than I first thought. In fact, a board of which I'm a member used an appreciative inquiry approach at a meeting last weekend in trying to shape a future vision for a non-profit organization.

So, when I think about expert judgment and the elicitation of that knowledge, there are principally two forms that we seek - generative judgments and evaluative judgments.

I definitely see the role of AI in the generative context where we are trying to have experts "generate" ideas, future states, scenarios, etc. Where I do not as clearly see AI having a role is in in the evaluative context where we ask experts to evaluate a specific quantity. For example, in project management, we often ask experts to quantify risks (and particularly the associated probabilities and distributions) for projects. Have you ever seen AI used to quantify such values?

Thanks again.

--Paul

Hi Paul,

when you say AI do you mean Appreciative Inquiry? If so, I'm afraid this is very different to AIM (Appreciative Inquiry Method) which is based on Geoffrey Vickers' ideas of appreciation and appreciative systems. It is not the commercial product advertised as AI.

Daune

Since you have expertise in AIM, that's what I was asking about. Sorry if I muddied the waters with my abbreviations. I'd like to hear if you thought AIM could be used in a context where experts are asked to estimate specific quantities (e.g. costs, times, etc.). Thanks.

Hi Paul, I think AIM can be used whenever you ask someone to explain and justify something. I would imagine you would use AIm if you wanted to try and find out how an expert thought they went about risk estimation (ie the process they engage in to reach their answer). I'm not sure how useful it would be to use AIM to generate the actual estimates. I hope this helps.

Daune