Again @Rauf is right, if you divide the groups as equally as possible and use the same test for each assessment then you are testing the increase in their ability from a known baseline and can see which method shows the greatest improvement.. It is not perfect but it does allow you the maximum control over many of the variables.

It is an interesting question but I think you need to decide exactly what effects you want to measure before you can decide how to measure them. What do you intend the integration of this technology to achieve?

Thank you sir. Well, I am into the teaching and learning process. In a conventional method, for example, we use the pre-post test method to measure learning gain. And if I will integrate blended teaching (Online) for experimental group and assuming other variables were controlled such as the topic, time frame, entry skills / abilities of students, can I measure the effect (learning gain) using also pre-post test method? Is there a standard assessment tool for that? Please advise.

Elmer Irene, i think applying the pre and post test method will be appropriate. The assessment tool will depend on exactly what you taught - subject, skills etc. For example, if i use blended method in addition to say, face to face teaching, or the other way round, to teach grammatical items, then i should use an assessment tool designed to test the knowledge of those items. Hope this helps.

As a short experiment you could measure acquisition and retention of new vocabulary, for example 20 or 30 words from the Academic Word List which is available online. This could be taught by both methods. You would use a pre learning test to determine the level of each student. Then after teaching the words and their meaning or use, you could give the same test to see how many new words the students have leant. A few weeks later you could set the test again to see how many words have been retained by the students. This is a relatively short experiment which should not interfere with the students’ learning but allow you to compare the two styles of teaching. A longer experiment could be used to teach a particular concept or idea from the subject they are studying. I am a language teacher so my suggestion is language based but I hope this helps to give you ideas.

I understand. Thank you so much for your valuable inputs @Gary Stobbs and @Mariyam Nashida.

Elmer Irene, I doubt if pre-test post-test method would be appropriate for the mentioned study. In such a scenario control group and experimental group would be the way to go. You will need to form two groups and keeping all other variables controlled teach one group by conventional means and apply the intervention (in this case technology assisted teaching) on the experimental group and record observations on the hypothesized objectives for later analysis.

@Rauf Azam you are correct I should have made it clear that the students needed to be divided into two groups then one would be taught using the traditional methods and one group taught using online methods. You could include a third group that uses blended method or even a fourth group that would use self directed methods. The main difficulty I can see with the study would be how to ensure that as many of the variables as possible could be controlled to give a true comparison.

@ Rauf and Gary Yes I am also thinking about that. But what would be the uniform entry skills requirement for the groups. Can I use their scores only in the pretest or do I have to control their intelligence or computer abilities, etc?

If your objective is to measure the learning outcomes in a particular subject (other than the computer skills) say Maths, Science or language then you may not need to worry about the computer skills. However, the knowledge of the participants in that chosen subject should be uniform across the groups which can be ensured through a pre-test.

Again @Rauf is right, if you divide the groups as equally as possible and use the same test for each assessment then you are testing the increase in their ability from a known baseline and can see which method shows the greatest improvement.. It is not perfect but it does allow you the maximum control over many of the variables.

A recent study titled "Effect of blended learning environment on transfer of learning: an experimental study" might help you think through this question. http://onlinelibrary.wiley.com/doi/10.1111/jcal.12009/abstract They also use pre- and post-tests.

It is important to note that only looking at the marks students acquire, might not be the only indicator of the learning that took place. A study by Stephenson, Brown and Griffin might help you think of other factors to consider. It is available at http://www.cblt.soton.ac.uk/multimedia/PDFs/comparison%20of%20electronic%20lecture%20delivery%20models%20university.pdf

I would return to @Gary Stobbs original question: "What do you intend the integration of this technology to achieve?" - adding not only "achieve in retrospect" but "achieve during the process". There are some crucial points I miss from the previous replies:

1. A pre/post-test design seems to imply that learning is something static, comparable to testing whether medicin A works better than placebo B. From the view on learning as a dynamic process which develops as you interact with the "medium" (ICT or person) you would like to define more specifically which ways you think your particular application of ICT (which of course also needs to be made more specific) will affect certain processing of information. There are too many ways you could frame this question to get into details here, but you might consider hypotheses relating to facilitating memory, increasing attention or engagement with the material, etc.

2. The point on "testing" leaves me with the concern that nothing has been said about how the test should be administered, which is of great importance when considering transfer. ICT develops too quickly and is too broad and variable for there to be any standardized testing forms, and so everyone doing these kind of studies should consider how the testing situation relates to the learning situation. An obvious point (from cognitive psychology) is about context-dependent memory: if learning takes place with a computer, isn't it more likely that students will perform better on a computerized test (in a similar environment), than if they have not been learning with a computer? Then you are not assessing the effect of instruction at all, but only the effect of the medium.

Rather than multiplying your control groups, I would suggest you consider measures of "learning" (however you choose to operationalize it) within the software itself, such as how one builds up skill within the same system, or when taking on new but related material, instead of testing what has already been processed. The concept of "Preparation for future Learning" (PFL) as a testing method might be informative for you, as well as the discussion of ICT's possible "added value". Stanford researchers Daniel Schwartz et al have done some excellent work on this.

Thank you @Elmer Irene for initating this interesting discussion and all you who replied - as a Ph.D student I am in working on similar issues and so far the "traditional methods vs ICT"-question appears to me as quite misleading. When using ICT you are changing BOTH the method of instruction AND the informational processing, because computers do not convey or "mediate" (construct) information the same way as people do (after all, there is the reason why we use computers at all!). I would personally appreciate more expert comments on this!

Thank you very much @ Björn Sjödén. Indeed it is not as simple as studying the effects of integrating technology (ICT in general) as compared to the conventional method. Adding one variable or simply changing the medium are not the sole factors causing the effect. @ Gary Stobbs, we both agreed of coming up of simple experiment, controlling some variables, limiting topics, and try the "cause and effect" design. If it is not as simple as this one goes, perhaps I need to consider other variables such as software, test administration etc. as suggested by Björn Sjödén.

We use open learning as one of the blend learning training methods. Students watch records and do their practical exercises individually while a trainer presents in a class. Trainer functions are help and control.

We use the following control methods:

- brief self-control quizzes or exercises during the video-lectires

- tests or labs or practical cases each 3 hours under trainer control;

- final test or case.

All results are collected and may be analyzed afterwards both by a student and a trainer.

@Elmer Irene is right that if the experiment is simple then we can have some control of the variables. That can give us some measurable results and would "prove" which is the "better" method but maybe this is not particularly helpful as @ Bjorn Sjoden points out the comparison between the two methods is a little misleading. The use of technology and online instruction is only likely to increase. It would therefore be more useful to determine the best use of technology and the best blend of traditional and online methods. We do need more research into the way that the method of instruction influences the learning process. How does the medium of instruction affect the learning process? How does the new technology available affect the experience? How does the method of testing affect the results? How does the style of the instruction (rather than the method) affect the cognitive processes behind learning? There are many more questions but the problem of controlling or at least understanding the variables is still an important issue.