Dear Ms. Mónica Garcia Melón and co-authors

Reference is made to you article

“Adapting RRI public engagement indicators to the Spanish scientific and innovation context: a participatory methodology based on AHP and content analysis”

Public engagement general definition: Term involving procedures by which citizens influence and take actions related to government decisions.

I believe that this definition helps understanding the purpose of your work. In your own words the purpose of the study is, as goal G 1,” to find relevant public engagement aspects from RRI perspective”.

I read your paper and my comments are:

1- You say “G4-To demonstrate that a participatory process with experienced stakeholders is suitable for the tailoring of the European indicators to the Spanish RRI context.”

I am afraid I don’t concur on this key point: In my opinion, it is incorrect to determine this participatory process using stakeholders. They don’t represent people, and they can’t vote or take decisions for them. This is clearly demonstrated in the Arrows’ Impossibility Theorem, where it is catalogued as dictatorship. Of course, if you use AHP, this method is based on the concept that a DM or a group of DMs can decide for others, something that I discussed with Thomas Saaty in 2017.

In page 6 you say:” Decisions made without enough social acceptance may be fragile, reducing the viability of the proposed alternatives (Grimble et al. 1997; Gutrich et al. 2005). This part of the study focused on a qualitative approach and a participatory initiative based on the opinions of these relevant actors (Ràfols 2019)”

This is absolutely true, but then, you contradict yourself. Where is the social acceptance in your methodology? The relevant actors are the people, not the stakeholders, and not even chosen by people as their representatives.

2- You say “Once the good performance of the methodology has been demonstrated, several recommendations will be inferred to use it in a more general (and official) context and with a greater number of societal actors.”

I wonder how you can demonstrate that good performance, if, common to all MCDM methods, there is nothing to compare to, i.e., there is no a benchmark or yardstick.

3- You say “The work we propose in this paper stems from the guidelines proposed in the document (Strand et al. 2015) where it is suggested the indicators proposed at European level be adapted to their respective countries”

I am in a complete agreement. It is very difficult, if not impossible, to analyze jointly diverse countries, with different needs, regulations and goals.

4- Is there any reason by which in Table 1 the Balearic, Canary Islands, Ceuta and Melilla are not represented? They have in excess of 3,300,000 inhabitants, about 7% of the Spanish population.

5- In my opinion, Public Engagement (PE) is related with many other aspects that need to be considered, like communication, education, social capital, social network, diffusion, technology inputs and output, etc., and each one of them with several sub indicators. Most important, all of them are heavily interrelated, and thus, they constitute a network with precedence and influences. Consequently, I believe that considering this scenario as a linear hierarchy is improper.

For instance, there a direct influence to social capital from trust, social inclusion, diffusion, collective actions and networks, society indicators, etc., with links in both ways. It is impossible by using experts to understand and analyze each one, let alone, quantify their importance, using a pair-wise comparison procedure.

I think that your approach does not contemplate reality.

I believe that the selection of twelfe stakeholders who represent diverse Spanish Regions, in indeed relevant, but not enough. Spain has a population of 47 million with a large variety of regions and with different levels in each one.

6 - Paths 1, 2 and 3 of Figure 1, are correct but not number 4, where you select a MCDM method that is unable to deal with a project of this nature and size, because, among others, the condition of a lineal hierarchy, when it is clear that the scenario is better represented by a network. Perhaps using ANP would have been a better option.

7- You say “If stakeholders get involved in the development of the indicators, they will become the “owners” of the monitoring, and therefore, they will more easily accept them as a valuable tool to improve their performance”

Hard to disagree with this concept, provided that the stakeholders are given assurance that the method is reliable, and, of course, assuming that there will not be preference or vested interest when monitoring. In addition, these policies take time to develop, and then, there is no guarantee that the same stakeholder that participated from the beginning, will be in the same position to monitor 10 or more years later.

8- You day “Decisions made without enough social acceptance may be fragile, reducing the viability of the proposed alternatives”

Very true. Now, who says that the opinion of the stakeholders reflects what the society think about an issue? Since they make decisions in behalf of the citizens, it looks more as personal decision than a collective decision.

9 - You say “AHP is a measurement theory based on the fact that the complexity of a multiple criteria evaluation problem can be solved through the construction of hierarchic structures consisting of a goal and several level”.

One goal? You mentioned four.

10- In Table 2

- "CB02 Degree of exposure to scientific information (media, social networks, science museums)"

How the stakeholder can measure it?

- Table 2 indicates “CB03 Horizontal + vertical participation in science”

How do you incorporate this vertical and horizontal indicator in AHP?

- “CB04 Public expectations of involvement”

How the stakeholder knows about this? Did he/she perform a survey?

- “CB06 Participation in networks (committees, associations, projects)”

This indicator corroborates what I mentioned above regarding the necessity of working with networks, not with a lineal hierarchy.

11- You say “Inconsistencies were checked, so that ratios were not higher than 0.1. (Saaty 1994). Two stakeholders (nr. 11 and 12) had inconsistency ratios higher than 30%, they were asked to reconsidertheir judgements but even after their second round,”

This means that stakeholder’s judgement that are assumed to be taken based on considering experience, knowledge and analysis, must be modified because a formula says so? Why to reconsider? Because they are outliers? Don’t you think that this procedure involves a strong bias?

In the same paragraph “the high level of inconsistency was maintained. Thus, for prioritizing matters, the judgements of those two experts were dismissed.”

As easy as that? Because two ‘rebels’ disagree, they are not longer considered? Remember that in this case the two stakeholders represent al most 17% of the stakeholders, therefore, their opinion and judgement is also important.

Why did you don’t accept them? This disagreement is not a problem when you calculate the mean of answers using the geometric mean, although probably, it could be an issue if you use the Eigen Value analysis.

These are my comments to you paper, and, if possible, I would like to have your response, even if it tells me that I am mistaken.

Thank you.

Best regards

Nolberto Munier