Ecologocial (or as it often is called alternatively "external") validity is a standard bone of contest in the experimental literature. Since economic experiments very often actually test a game, in this literature one regularly finds discussion of this point.

The reference is, admittedly, parochial. But you might find one my papers useful where I discuss the comparative advantages and disadvantages of experiments for behavioral law and economics:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2276566

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2207921

Thank you Christoph, I look forward to reading your papers.

I appreciate Ecological Validity is a very charged term, and I am not too sure as to how well I am wielding it. I have traced it back to Brunswik, mainly following texts at bruswik.org and particularly those of Kenneth Hammond. I am writing an MSc paper about some research we did to create a health game for potential use by forensic mental health services. We spent a long time with the service-users, understanding their life and behaviour in order to simulate this in the game. It is this simulation that I am referring to as '(a degree of) ecological validity' - I am open to correction!

Paul,

Ultimately, the only faithful description of reality is reality. But if you just leave reality as is, you are at the mercy of naturally occurring random variation. If mother Nature has not done you the favour of randomly exposing a sufficiently large portion of independently acting individuals to the intervention you are interested in, and a sufficiently large portion that otherwise is identical left unaffected, you cannot make causal claims. This is why we artificially create randomness - in your case with a device that you believe captures the essence of the effect out there in the world you are interested in.

I think the most useful way of dealing with the issue is what economists tend to call "identifying assumptions". By the very fact that you artificially intervene, it is clear that what you study is only analogous to what you want to understand. Discuss these differences, and carefully argue why they are not critical for the lessons you want to draw from your results.

Quantification of this distance seems difficult to me. It would require that you generate some form of a proximity index. Yet the reasons why what you study differs from what you want to explain are bound to be qualitative, and thereby multidimensional.

Hope that helps

Christoph

Thank you again Christoph. If you have time, can you address the following,

If I understand you, true Ecological Validity will only ever be found in the fist ecology. Therefore, on a very philosophical level, even a perfect recreation of a given ecology can only ever be deemed ‘approximate’ in respect to the original (the 2nd butterfly just might slightly flap a wing out of time with the 1st). Further, what can be achieved within the context of a digital game is obviously light-years away from any perfect recreation. Consequently, in experimental terms, we can only contrive a device that "captures the essence of the effect out there in the world you are interested in".

As you mention, the inadequacies start with our game being based on a very-selective interpretation of conversations with a handful of service-users. We then try and recreate a very skewed sub-set of this dialogue using polygons, Photoshop textures and C# - certainly not the primary constituents from which the users normal ecology is constructed. In-turn this conceit is then evaluated by a narrow sample of players. Thus, I believe, you are saying it is important specifically acknowledge these limitations, and also to clearly explain what has been included and why ("identifying assumptions").

In your opinion, given that the limitations-of and reasons-for using device 'x' are explained, is it reasonable to then state "doing 'x' in the game helped achieve a greater degree of ecological validly", with a measure of this 'ecological validity' being based first on existing interventions used in the domain and then on feedback from players of the game (all essentially qualitative, as you state)?

Thank you again, and apologies for any slow comprehension on my behalf.

Dear Paul,

No reasons for apologies whatsoever. You are at the heart of empirical methodology. As economists tend to say: there is no free lunch. No empirical evidence is perfect. But that is by no means a justification for not trying. Ultimately the best approach is what some empiricists call triangulation: combine alternative empirical methods, and see whether results (reasonably) converge.

For your concrete project, you come close to what some economists tend to call the "field in the lab" approach:  you invite participants who know that they participate in an experiment. But you present them with a setting that tries to capture multiple features of the real life setting you want to understand. The approach has the advantage of being closer to the ultimate object of study (higher external validity), but ultimately you may never be sure whether your manipulation has caused a treatment effect, or the interaction of treatment with one of the features of your richer design (which reduces internal validity). There is nothing wrong with this approach. Just acknowledge it as a deliberate choice, and if you have a hunch about possible interactions, discuss them.

All  the best with your project

Christoph

Thank you Christoph, very helpful.

My focus is on the "ecological validity" aspects of the user interface, with respect to games.  Under the "Variety of Realism" section of the article: "Enhancing Realism in Desktop Interfaces for Dismounted Infantry Simulation", we make a distinction between physical realism & behavioral realism.  I suspect that behavioral realism comes close to what you mean by ecological validity, at least in terms of training transfer.

Good luck with knitting all the different aspects together.

-JimT

Conference Paper Enhancing Realism in Desktop Interfaces for Dismounted Infan...

Hi James,

Thank you for the input and apologies for my slow response – I missed any update from ResearchGate, sure they was one, they tend not to miss an opportunity to mail =]

The first-hand observations you did make an intriguing juxtaposition with the requirement's gathering process we used. We had to go with conversational descriptions, from volunteers in a secure setting, of their street drug-use behaviour. We were mainly looking to understand believable locations, believable characters and what they talk like - the player currently navigates in 1st-person/as-a-camera perspective. (Interestingly, on evaluating the prototype game, we got some unsolicited feedback from the volunteers on how the game-characters ‘moved’ realistically, menacing like a drug-dealer – I had just slapped-on random stock animations, time being against us!)

Thank you for pointing out Physical & Behavioural realism, this has opened up a new stream of investigation.

EV is a difficult concept at the best of times - and fitting it to simulations and serious games, is particularly spikey. In this regard I have found, flow, presence, immersion and affordance, concepts with a larger, and less contentious, body of empirical research.

Here are a few additional items that may be of use to anyone following this thread, the links are to external pages,

Frameworks to evaluate simulations,

1. van den Hoogen, J. et al;"The Debriefing of Research Games: A Structured Approach for the Validation of Gaming Simulation Outcomes", http://informs-sim.org/wsc14papers/includes/files/312.pdf

2. Peters, V.A.M. et al, 2004, "A simple classification model for debriefing simulation games", http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.195.1715&rep=rep1&type=pdf

3. Graafland, M. et al, 2014, "How to systematically assess serious games applied to health care", http://games.jmir.org/2014/2/e11/

Interesting interventions in Forensic Mental Health,

4. Playmancer, https://www.youtube.com/watch?v=osmo9EAClv8

5.  Arborelius, L., et al, 2013, "A new interactive computer simulation system for violence risk assessment of mentally disordered violent offenders", Criminal Behaviour and Mental Health, vol. 23, no. 1, pp. 30-40.

& number of works from the following authors, Tobias Greitemeyer & Silvia Osswald, Giuseppe Riva, Laura Ermi & Frans Mäyrä.