AHP ratings are checked for consistency.

Quoting a paper done by Saaty and published on "How Many Judges should be in a group?" that's how many respondents are enough, he concluded

"To engage judges to help with a decision should not be a random matter. One needs to know the area of expertise needed to make that decision and select a judge or judges that have both knowledge and practical experience with the matter. In this case one expert judge may suffice unless political expediency requires that several judges from different constituencies are necessary. In that case one might select several judges if they are available."

Follow this link attched

Article How Many Judges Should There Be in a Group ?

Thank you Peter Mwangi Macharia Sir for your valuable answer.

The Analytic Hierarchy Process (AHP) is a theory of measurement through pairwise comparisons and relies on the judgements of experts to derive priority scales. It is these scales that measure intangibles in relative terms. The comparisons are made using a scale of absolute judgements that represents, how much more, one element dominates another with respect to a given attribute. The judgements may be inconsistent, and how to measure inconsistency and improve the judgements, when possible to obtain better consistency is a concern of the AHP. The derived priority scales are synthesised by multiplying them by the priority of their parent nodes and adding for all such nodes. 

http://www.colorado.edu/geography/leyk/geog_5113/readings/saaty_2008.pdf

 I set out to know how different experts in professional homogeneous group vary in terms of their responses. It maybe necessary to have many experts but if they do not make informed choices based on their professional knowledge its futile to ask the many. See the attached link

Article Experts’ Responses Comparison in a GIS-AHP Oil Pipeline Rout...

Thank you Dr Krishnan Umachandran; does this mean checking for consistency is enough for validation?

 Thank you Peter Mwangi Macharia so much.

In Analytic Hierarchy Process, we use step wise process.

Everyone knows what is AHP i.e. one of the MCDA. Experts opinions should be professional and expertise in the area. There should be open house panel discussion for validating the responses.

About the validity of rating can not be same for a person. It may varies from person to person opinion. Thus, as I have mentioned earlier that all experts should be professional and expertise in the area.

• Choice problem
• Sorting problem 
• Ranking problem
• Description problem 
• Elimination problem
• Design problem

The Analytic Hierarchy Process (AHP) is one of these MCDA methods. It had been developed for ranking problems and occasionally for choice problems. Whilst other MCDA methods have evolved in order to be applied in sorting problems.

The analytic hierarchy process (AHP) is a --- structured technique for organizing and analyzing complex decisions, --- based on mathematics and psychology ---has particular application in group decision making, and --- is used in a wide variety of decision situations, in fields such as government, business, industry, healthcare, shipbuilding and education --- is most useful where teams of people are working on complex problems, especially those with high stakes, involving human perceptions and judgments, whose resolutions have long-term repercussions --- has unique advantages when important elements of the decision are difficult to quantify or compare, --- where communication among team members is impeded by their different specializations, terminologies, or perspectives --- etc.

Decision situations to which the AHP can be applied include: 

Choice – The selection of one alternative from a given set of alternatives

Ranking – Putting a set of alternatives in order from most to least desirable

Prioritization – Determining the relative merit of members of a set of alternatives

Resource allocation – Apportioning resources among a set of alternatives

Benchmarking – Comparing the processes in one's own organization with those of other best-of-breed organizations

Quality management – Dealing with the multidimensional aspects of quality and quality improvement

Conflict resolution – Settling disputes between parties with apparently incompatible goals or positions

http://en.wikipedia.org/wiki/Analytic_hierarchy_process

Thank you Barbara Sawicka for your valuable answer.

Thank you Subhash C. Kundu Sir for your valuable contributions.

Thank you Krishnan Umachandran Sir for your valuable contributions towards providing information about types of problems.

 Thank you Rupesh Kumar Sir  so much for your valuable contribution.

AHP has been widely used in different context and it seems there is no big issue with its validity. The validity of the findings using an industrial case would be helpful but is very time consuming as validating the results would

require comparing two cases, one that took into account the factors ranked in the study and the other that did not, to demonstrate that relying on importance of the factors actually improves the performance of a system.

A second point I would like to share is that, I personally think that AHP would rank the factors and shows their importance but can not rank them as critical factors. To find out the factors are critical, we can use other techniques such as ISM and/or Fuzzy DEMATEL to see if they are cause factors as well or not.

Thank you very much Farhad Panahifar Sir for putting more light on the issue of validation.

AHP methodology incorporates eigen values validation