Rob,

I am skeptical on several grounds that leads to the AI described in your question.

Thus to summarize on whether our human values taken to the extreme leads to such strong AI my response would be no. Several other valuations such as short term economics weigh in. Even if these were overcome my view is that strong AI would seek to implement an epistemological agent rather than an axiological one seeking value judgments.

My biased opinion

yes, thats right i agree

Your answer is well-informed but deals with whether it “will” happen versus if it “ought” to happen. I specifically didn’t ask whether it will happen, because I wanted to avoid the laundry list of confounding factors, of which there are many more than the ones you listed. That last part, though, about your preference of an epistemological strong AI versus a value-assigning one sort of answers the question.

The axiological priority is based on the idea that without a value-assignor, value can’t exist. Hypothetical: IF the human race died out and left behind a sentient and sapient strong AI, would you want it to be epistemological or axiological? (I’m making this either/or on purpose.) And if epistemological, where is the value in that, absent an axiological point of view? I hope this gets you closer to what I was after.

What if you could make an AGI that is as capable of rational or irrational thought as much as any human? Personally I think its best to put aside everyone concerns and just plow ahead with AGI research... nothing is more important in my mind...

Rob Lubow ,

I find your deontic position curious based on the contents of your question and your constraints for it (it is like asking a physicist to contemplates Lewis' tailless Kangaroos) .

Anyway, to answer your more constrained question on why my choice of epistemological AI, it goes as follows:

If I base my AI's goals on having a justified true belief(let's put aside Gettier's problem for which I have a particular response), then the objective itself is knowledge with those properties such that in the end it can answer the big questions of why to itself (or other beings).

An AI that is based on value judgements will classify things as good and how good they are (to itself and other beings), but will not disseminate knowledge.

In the end asking and be able to answer such questions that we have forgotten as to why we are here would be my ideal goal for any AI.

As a side note: an AI based on value judgements will be more difficult to reason with if gone wrong than an epistemological one. Wrong programming based on an assumption that what he knows is reality as opposed to an epistemological "this is what I know" can be really dangerous (on your assumption that we achieve this AI). This is the usual imagery of what people have when they describe a rogue AI.

I think one issue is that its was not a " AI based on value judgement " but and AI based on the assumption that intelligence that can assign value and are sapient and sentiment have value". at least with the ICOM program that Rob is referring to it is not the values assigned that is important but the preservation of the sapient and sentient intelligence's. In theory the ICOM system is a system that is based on emotional subjective experience biased to experience ethic's and morals the way a human would with the same sort of potential for emotional problems even. at least until we understand it better.

Right. I don’t think we need to choose between an intelligence programmed for just “is” or just “ought.” It should have both, but the question is about the basis of the “ought“ for an AGI. This seems we’d need a universal, scientific basis for morals, ethics and values, but we don’t have anything like that. So how do we imbue AGI with a sense of “what matters“ in such a way that it makes decisions in line with humanity’s greater moral compass? And to what extent does that even matter?

David Kelley

If the ICOM program is being specifically used for the question it should have been explicitly stated. Since it was generally stated and not refocused by Rob Lubow I will maintain my position that value judgements do not need to rely on intelligence as refocused on the either or selection mentioned afterwards. It is something that does not follow from the current definition of AI. You are assuming all AI is just symbolic AI which is not (there is algorithmic optimization and pattern recognition which do not entail intelligence, just the recognition of a stored pattern or fulfilling an optimization function).

Rob Lubow

I do not think there is a need for a scientific basis of morals, please verify the project by Hartman on value, I think he does a good groundwork on which to base value .

Also, I think I answered the question in the negative of " whether our human values taken to the extreme lead to creation of the strong AI described above", the answer is no, it can be epistemological and not axiological.

Finally, your assumption "decisions in line with humanity’s greater moral compass" is not achievable because most prescriptive projects on axiology would not fit with that assumption when programmed in a symbolic system. Even if we program a learning algorithm for a descriptive approach there is the Bias/.Variance dilemma of AI to be overcome in this arena. Again, I am assuming generality since no specific restriction has been imposed on the axiological scope beside a very broad statement.

I've submitted a paper on a similar issue to Paladin (the topic is Metalaw and AI).

I would be skeptical when taking into account the base from which the AGI is formed. It is either fed information (trained) or programmed from scratch. The problem comes with both imputs having human cultural elements, which in fact may bias it toward some assumptions.

Self-sustainability and self-developement all can stumble upon notions that the isolated AGI cannot refference without building its own system of knowledge, understanding and assumptions. Until we create a machine that can reason on its own merits, without referncing anthropic philosphers, then we can be closer to the anwser.