I suggest that such a system can only be evaluated with a concrete task that allows to quantify success. This also involves considering the embodiment of the system. So in that sense the competitions you mention seem useful. However, typically you would want your cognitive architecture to be as general as possible and not limited to one specific task; this should be considered setting up a set of evaluation tasks. Comparison to empirical results from humans seems useful if you either (1) want to mimic their behavior (e.g. in some HCI situation) or (2) if you explicitly want to model human cognitive system (i.e. to make predictions how a person would act in some task). The task and embodiment should be kept in mind. There is no use e.g. to compare human gaze-behavior with that of a proposed 6-armed and 3-eyed robot that catches flies out of the air --- nor to judge a human in fly catching task.

I agree that the question of evaluating such systems is not sufficiently addressed yet. Often the definition of a target system and task is missing and comparisons are made with ground-truth from human subjects whose behavior is necessarily influences by several cognize states that are not part of the evaluated system (e.g. influence of knowledge). I think it would be an advancement if hierarchical classifications* would be created for such systems with tasks and metrics provided and assigned to the classes. A system can perform different tasks of the lower hierarchy levels and it can be compared with specialized solutions on the lower levels and with other systems on the same level.

*e.g.: System-that-can-do-everything

Thanks Jan. What do you suggest as evaluation tasks? Is the definition of use and test cases with pre and post-conditions etc (as in software engineering) and testing the system (in a simulation) if it fulfills the requirements a appropriate way to evaluate it? E.g. is an agent able to survive in an artificial life simulation.

I am afraid that such a method of evaluation would lead to task models rather than process models, which should be approached for a cognitive architecture.

One way to evaluate cognitive agent architectures is to identify their capabilities, and compare these to both theory and empirical research in cognitive and behavioral psychology. This can help answer questions regarding HOW agents will accomplish certain tasks in certain environments. Of course, it doesn't answer how well they will perform. For example, if the agent architecture is focused on Belief-Intention-Action (BIA) reasoning using static production rules, it would be inappropriate to use it in experiments where you are evaluating agent creativity (formulating and using novel concepts).

Dear Samer and Russell,

I agree with Russell,

The best cognitive agent is human brain, and the best approach is to study brain-mind and model it. Of course BIA is not a valid model of the mind. I do not think it is even worth mentioning.

The Agent paradigm is only an idea for a cognitive model and as stated by Leonid there is only one valid example - and we don't have a model for that yet.

Imagine that all of your cognitive aqents have a shared language (assumption: they all share the same environment) if you can get them to produce plausible discourse about their environment and it is understood and used by the others then you getting somewhere. Note discourse production is not the same a pre-planned/defined communications.

To solve this problem you will need to make their environment very rich and totally consistent. An then you need in internalised model of that environment within your agent.

Note: this problem has nothing to do with the Turing test - the key is the degree of coupling between agent and the environment and its internalisation as a model.

Thank you John Sanders for clarifying this. My response referred to rather concrete cognitive systems (probably not what Samer Schaat meant) but they do share the problem of evaluability to some degree. I'm not familiar with he idea of the agent paradigm but wouldn't you say---when you move from the idealized model to practical systems---that the concrete tasks, environments and embodiment are of crucial importance for the evaluation (which must not necessarily agree with those of humans)? Even if a universal model is the goal still progress must be made measurable somehow.

Dear John and Jan,

I agree. I would add that the best evaluation is practical use in the intended environment. It might also be easier than trying to simulate and adequate test.