Hello Yusuf, yes you can you're correct in all the analyses performed. Best wishes David Booth

David Eugene Booth, thank you so much for feedback and the kind wishes.

You claim "There is no difference between students who receive the treatment and those who does not receive the treatment in terms of their motivation." How are you claiming that there is no difference between the students? Do you mean randomization was used and the expected values are the same?

bu

bu

bu

bu

bu

bi

Daniel Wright, thank you very much for the reply.

As for your question; since the study was carried out at K-12 level schools, complete randomization was impossible. However, we used convenience sampling. Before the study began, we used a pretest and observed that there was no significance in the mean scores of both groups of students (Experiment and Control).

Best regards.

Yusuf Polat , that does not mean that the groups are the same (or even very similar given the sample size), even on that measure. Without randomization it becomes more difficult to justify just using the post measures. Further, since this was in schools, have you accounted for classroom effects (i.e., two kids in the same classroom are not independent)?

Daniel Wright, I haven't done accounted for the classroom effects. However, since both of the groups (Experiment and Control) were already in different classrooms, I assumed the pretest and posttest design would be enough to measure the effect of the treatment.

Thank you once again.

@Daniel I am told by educational psychology people that this is a standard design. Further that it fits as a quasi-experimental design. I have long since given up on pointing these things out since there's really nothing to be done about them given current constraints. I'm happy enough seeing that people are attempting to answer useful questions in a scientific way. If this is indeed the best that can be done let's minimize hassling the researcher after s/he has done their best. Best wishes, David

If you only have two classrooms, then at this level you have only n=2. This situation was one of the main contexts that helped multilevel modeling take off (see, for example, http://www.bristol.ac.uk/cmm/, at the college of education at U of Bristol). It is worth looking at the resources on this page for this issue. As far as quasi-experimental designs and making inference just from the post scores, there are numerous textbooks (e.g., https://www.amazon.com/Counterfactuals-Causal-Inference-Principles-Analytical/dp/1107694167/ref=pd_lpo_1). David Eugene Booth , if you think "all the analysis performed" are correct, as you say, well, you are welcome to your beliefs.

@Daniel you are correct in your last assertion. However in this case we are interested in students not in classrooms. Wishing you all the best, David PS my belief is backed by a record that you may at your leisure examine. Please note that I have been plagiarised heavily by an individual. I ask that you consider only the record on RG that is made of papers that I control the listing for. These are my featured papers. By the way RG refuses to delist papers supposedly authored by me that I request. Any questions can be referred to Executive Director American Statistical Association who sanctioned the offender. Best wishes, David

@Daniel one last point would you please explain what you would have done differently if you had been in Yusuf's position.? Best wishes once again David PS criticism is easy research design is not always as easy.

David Eugene Booth , I do not know exactly what Yusuf Polat has done. He has asked several questions on RG over the last few days. It would be good to see if a power analysis or some other justification for the sample size was taken. If classes are to be the unit that are assigned to conditions, this should be taken into account. If whatever method to guide sample size choice produces values that are not possible, alternative designs should be used. But, starting with the non-equivalent group design, something that your ed psych friends presumably know about, you would use the pre test and the choice is usually how you account for the non-equivalence (e.g., gain scores, ANCOVA). Lord was at ETS much of his career, so your ed psych friends would also know about Lord's paradox. My guess is that the sample size may be too small, but I do not know how the sample size was decided upon. As many people (e.g., Gelman) have argued small samples, when significant, can produce effects in the wrong directions and errors in the magnitude of the estimates (Type S and M errors, respectively). From what I can tell I think my suggestion would be to plan future studies carefully.

Daniel and Metin, Have you never seen a real life problem away from a dissertation defense. Yusuf had one and he did a good job with it. Does it possibly have errors sure it could. That's what a statistician lives with. He used all of his information the way that he had to. I never got to Choose who or how many students I had in a class did you? A power analysis in a grade school setting surely you jest. You two are out of touch with reality if you think that's what every

statistician can do every time. Yusuf you did a fine job with what you had to work with. Don't let these two inexperienced with real world problems individuals put you down. Type M and S errors for heavens sake I have been an ASA member for almost 50 years one of the first group to be accredited and I haven't heard of these either. Yusuf did something really something that few in his position would have thought about and he did it well. Congratulations to you Yusuf continue to learn and be a credit to your profession Remember that there's always a chance of an error in statistical analysis try to protect against them when you can. And remember that nobody ever knows more than what some professors think they do. It usually means that they have not faced that problem before. D. Booth

I guess I need to get more experience in the field of educational psychology!

Daniel perhaps you need to visit a grade school where this kind of situation is likely to arise . And realize that not everyone needs to do dissertation level research. After all it was John Tukey who said at a consulting visit please give me a ream of graph paper and not let's do a power analysis . D. Booth

Thanks for the suggestion David. I just wish I could find folks in educational psychology so I could do that. Apparently they do that.