This paper should help answer your question

Depending on the research I have used different methods:

For one of my theses I used a second layer data capture software. The software was created especially for the analysis and capture data in a processual way which let me make some kind of analysis like clustering and Bayes networks.

For other theses and works I have used forms, structured interviews, chronometers and activity counters (an action was done or not? how many times?).

I have also worked with people that use map tracking techniques as a second layer data capture software to make graphs analyses.

The trick is to centralize the database so that the data goes to your database or to create a saving file that can be mailed to you or that you can retrieve from other computers at some point.

It all really depends on what you are researching. I made once a research about social representations in video games and of course the measurement had to be made in a totally different way.

I hope the answer is useful for you.

What I am going to try in autumn and winter with a group of students is to let them play while using Camtasia, a software that is able to capture both a screencast (of the game, in this case) and via second channel a webcam centered on the player , combined in one video file. We are going to use this for a hands-on training in qualitative research on the games in question, though, but I suspect that if you find a way to run another software over the video files in search of activity metrics - eye movement, blinking, head movements or such - this could be used as a quantitative tool, too. You would only have to make sure that players play the same game in similar circumstances for the same amount of time to generate a reliable data set. And of course you would have to find a way to transfer the videos to your database, maybe via uploading them to a webserver, when the players shall not be in the same location as you are.

If you do find a way to use these files quantitatively, there is also a lot of Youtube-content already that could perhaps match your criteria, that was part of what gave me the initial idea (and the paper of a colleague who argued it could be turned into a method). Lots of horror game players use similar techniques to film their reaction to particular scary game scenes and to upload that on Youtube so that you can both see the scene in question and the shock it gave them. But I suspect it would be difficult to establish the overall comparability of such a sample.

I use what is called a play aloud protocol analysis

Play aloud protocols are very used, but personally I find that kind of protocol work good for usability and QA testing. Nevertheless for psychological data collection they tend to distract the player from the task adding extra cognitive charge so you cannot be sure of the variable you are measuring. In other words, you might be adding an intervening variable to your study.

depends upon your test and the dependent variable Carlos.

Yes, that's why I said "you might", because it all depends of the variables you are measuring and the kind of task. Also how well trained are the study subjects in play aloud protocols, not all people achieve a good performance in play aloud protocols so your sample should be chosen carefully.

Brock, I want to ask you, do you use play aloud protocols for quantitative data? How do you isolate quantitative data from verbal protocols, using discourse analysis? This really interest me, maybe I can find other uses for that kind of protocols.

Thanks for all the responses, very helpful. However, I'm still trying to understand remote data collection, as in I as the researcher would never see or interact with the participant in person ever. This is used in Amazon Mechanical Turk, but is somewhat limited to surveys. This might require more of a programming answer to implement a backend feature to an app/game that would document and track certain behavioral and game metrics information based on what the participant does within the app/game, hopefully without their knowledge.

Lauren, that's pretty much how I carried out my researches. But you should be aware of two things: the data collection software normally must be designed especially for the purpose, and that is expensive. Second, the user has to sign a contract in which you state that you are gathering their data for using it in research. I do not know where are you working, but in Colombia and Denmark (where I have worked) it is illegal to gather data without a consent from the participants. The consent can be done in an easy way (license wise), and the database can be done using cookies and synchronizing with the cloud. Obviously those kinds of details must be spoken with the programmer.

If you're looking to track player-generated control data or motion of actors in game-space over time, we use a customized version of the Unreal Development Kit (UDK) which outputs actor motion and action over UDP using the Open Sound Control protocol. We then capture the resultant data stream using a simple ruby script and write it to file. The data is then uploaded via shell script to our servers for analysis.

Depending on what you're looking to do, a fully rendered Unreal Engine implementation might be overkill. If not, the project is called UDKOSC and can be downloaded from Git at https://github.com/robertkhamilton/udkosc

I asked a similar question to myself in 2006, and wasn't able to find a 'key word' to describe this remote data collection method, so I coined one for my research, 'Information Trails'. In 2010, Zoeller's presented how he used Dragon Age to track players data (using a company server to collect data of players), and he used the term 'telemetry'. If you search Google using this term, you will find that computer scientists have pretty much picked up on this term.

Zoeller has a chapter in Seif El Nasr's new book "Game Analytics" on the same telemetry. I believe the book is useful in your case.

If you are interested in "Information Trails" you can find my papers by looking me up in Research Gate.

My research Lab created an Information Trails (telemetry) tracking system used Neverwinter Nights 2 as the game engine. (we are looking towards expanding, but progress is slow, because the NWN2 system works very well... if not broken, why fix?)

If you don't mind the medieval limitation to your 'game story', and are interested in simply learning how to collect data remotely, you are welcome to send me an email with more specific questions.

Thanks for the answers and the good info, everyone!