Perhaps this gives you some ideas:

Computational anatomy and neuropsychiatric disease: probabilistic

assessment of variation and statistical inference of group difference,

hemispheric asymmetry, and time-dependent change

John G. Csernansky et al.

NeuroImage 23 (2004) S56–S68;

E-mail address:

[email protected] (J.G. Csernansky).Available online on ScienceDirect (www.sciencedirect.com.)

@Béatrice - Thanks for your response.  I was looking for something more in the nature of human interaction, rather than anatomical.

Thank you for the question. Looking forward for more responses.

So you will be observing (participant observation, or not?) human interaction (verbal, non-verbal) during a workshop aimed at teaching discussion techniques, I assume. Are you also the teacher/trainer? Then this may become extremely demanding and complicated. But probably you will be supported by neutral, non-participating observers, or you might be videorecording the sesssions (=workshops) and do videoanalysis later on (time consuming, but probably the most rewarding technique).

Did you choose the topics for discussion already? Did you choose your discussants already? Students? Novices? Experts?

Do you have specific hypotheses or research questions already? What are the 3 basic learning goals? You will need to be explicit here, so that you will be able to select appropriate learning indicators: verbal, non-verbal, pairwise, groupwise, memory use. Is it important that group members learn a technique, which they can consciously talk about (meta-learning)? Etc.

Observation only will not be enough. This kind of learning is largely going off tacitly, internally. So you will have to ask, probe by questions, call it mini-tests if you like. Those tests - of let's say 5-10 minutes maximum, will be spread out over the duration of each session. The tests will be part of the learning, as you indicate, so it doesn't matter if some metalearning transfer between tests occurs. The items will be on a much lower level than the "items" you mentioned in your question. They should be easy enough to answer quickly and robust enough to indicate learning progress if there is some. 

Experimental design is more important than the measuring instrument. Good design will show you what you can feasibly teach, train, exercise and measure, cyclically. Don't forget to do at least one pilot study to validate your design set-up.

There seems to be one or more traditions of empirical research, at least I found this (below) review of 45 years jury decision making research. But not all discussions will end up in decisions; if they do, there is a concrete outcome of the process, which you may compare between groups.

Good luck.

another interesting lead may be: Focus Groups, well-established technique in management and design studies. Of course: brainstorming is another well-known technique with its own research tradition.

@Paul - Thank you for reminding me of the Devine (2000) study in this context. and for the focus groups and brainstorming research leads.

You are correct that I will be a facilitator at a test session of a workshop, and I will have at least one co-facilitator, depending on group size.  Participants will be expert in their field, but less than expert in the workshop topic, which is a pedagogy method.  One of my design objectives is to create documentation that allows other facilitators to present the workshop in a consistent manner at various times and places.

Your excellent advice reminded me to think like a researcher in designing the instrument rather than as a facilitator.  In terms of measuring attainment of the learning objectives, I have already included several written assessment methods which are, indeed, part of the learning activities.   I am thinking the group discussion instrument follows a qualitative method, just as you suggest.  Looking at it that way, what I am trying to do is identify in advance the themes or issues that I think will emerge, and wish to advance, while leaving space for the facilitator/observer to also document discussion themes that emerge, that is, themes that were not predicted.  From the facilitator's point of view I am also trying to provide a checklist (as it were) of themes that signal the cohort's progress towards readiness for summative assessment and/or progression towards the next objective level.  What are your thoughts?

Aha, now I understand much better what you are after. Actually, your primary subjects - speaking experimenter jargon - are not the workshop participants, who rather act as part of the process being facilitated and observed. Instead future facilitators are the primary subjects whose effectiveness you want to know and optimize.

My question would be, whether you can do that by just imagining how a (perfect) facilitator should work, placing yourself in their role and observing yourself while actually doing their work. For a first pilot, that might be OK, to "get the hang of it", but sooner or later you might need to stop role-playing, get out of the loop and concentrate on close observation of other discussion facilitators in order to identify the main control (facilitation) variables in such situations.

Indeed, this is a rather complicated empirical design involving two closely coupled processes - discussion versus moderation - where the success of both depends upon in-depth knowledge of both processes on part of the moderator aka facilitator. Guess you should prepare for a long journey ...

I think that Richard Anderson's work around studying collaborative reasoning might be useful here. His paper "The Snowball Phenomenon: Spread of Ways of Talking and Ways of Thinking Across Groups of Children" presents key data but also reviews other important studies in the field. 

http://www.tandfonline.com/doi/abs/10.1207/S1532690XCI1901_1#.VFtnSVOsWdk

@Colin - thank you. That and another subsequent paper by Richard Anderson got me moving towards the area of interaction analysis. I have found some lines of inquiry involving analysis of asynchronous discussion text for distributed learning and meaning-making.  I am looking at these for concepts/methods that can be applied in a live discussion context as well as subsequent recorded media review.

Although I don't know and haven't yet applied the method myself, some friends of mine continue to recommend so-called Grounded Theory, which is a method, not a theory. Actually, it is a way to proceed in the so-called 'discovery' phase of scientific research, especially social science research. Should be useful whenever you still don't have a well-developed theory of your research object(s), so hypothesis testing is out of the question.

There are some tools to help you sort out what you observe, the best one (apparently) being Qualitative Data Analysis with ATLAS.ti (currently reading the book, see link). Helps with building a network of concepts, methods, behaviors, rules, tentative hypotheses, and a lot more.

http://www.amazon.com/dp/1848609930

http://www.amazon.com/dp/144628204X

Anna Sfard has proposed a scheme of discourse analysis, to characterize the change of discourse (learning) in a community of practice.

Sfard, A. (2007). When the Rules of Discourse Change, but Nobody Tells You: Making Sense of Mathematics Learning From a Commognitive Standpoint. Journal of the Learning Sciences, 16(4), 565-613.

Sfard, A. (2008). Thinking as communicating: human development, the growth of discourse, and mathematizing. Cambridge, UK: Cambridge University Press.

@Paul -- Thanks!  I have wondered whether this might be a good time to put Grounded Theory to work.  I have also wondered about current tools and look forward to exploring.  It looks like the Mills book will be a fantastic resource to go beyond the treatment in Creswell's  Educational research 4th ed. text: 978-0131367395.

@José - Thank you! I have added these to my reading list - they look very interesting. I see that Anna Sfard has made her 2007 paper available on RG and have downloaded.

I have studied the use of scoring rubric in conversation classes. I've designed the rubric according to the criterion of speaking skill the majority of conversation teachers depend upon in students assessment. 

If you are interested, please check my thesis (MA) or article. 

What are your personal thoughts about the cost-effectiveness of scoring rubrics? Could your research results have been achieved - in principle - by some other scoring technique - or do you think (on the basis of your current experience) that scoring rubrics have a unique potential other techniques lack?