Dear Ender Araujo

I think your question points on the essence of statistical education research.

I believe it is useful however, we should not neglect the importance of 'triangulation'!

Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education. London: Routledge.

It depends on how you will measure this improvement. You can use qualitative methods in order to describe the impact of the improvement. But, you can use statistical methods if you want to see this improvement through the scores of the students in the tests. Most of the times, both qualitative and quantitative research is the most appropriate type of analysis.

You may want to review the approaches taken in various on-line learning tools across the STEM disciplines that you quote. One approach seems to be to direct students to master concepts and principles by asking them to perform random yet systematic assessment questions. Another approach is to design lectures "on the fly" based on interactive responses to in-class anonymous questions, usually using clicker inputs. These are both designed to improve students performance in real time as the class progresses.

Dear Weimer,

If you have bibliographical references available, I would appreciate it.

Best regards,

Ender

My comment is in one case based from the sidelines as I listen to my colleagues who teach our 100 level chemistry courses. Since I do not teach them, I do not know what specific learning management system we use. I also sometimes see references to teaching and learning methods as I skim various reading sources. I don't keep track of these resources or if I do, it is rather scattered.

In summary ... I have no bibliography to provide. A better starting point to collect this type of information may be to talk directly to faculty who teach first year chemistry, physics, and math courses.

I believe that it makes sense to perform a statistical analysis to evaluate how to improve the learning process and the teaching process during teaching. First, inferential analysis provides additional confidence to your qualitative research outcomes. Second, it is a useful way to evaluate your study prior to classroom implementation.

Doing a statistic analysis on the student answers to test questions can tell you what portions of the curriculum that the students are not grasping well. This can be used as a tool to strengthen those areas in you lessons.

Yes. It is necessary to use statistical analysis in such a study. But after testing for the significance of any observed difference in the group means of the groups compared, you should go further to assess the effect size. It is from the value of the effect size you can draw an implication of the result for practice. You may go further to determine the integrity of the effect size before drawing the implication of the result.

Not very useful considering the massive effort to gather relevant and reliable data and make sense out of such statistics. There are other avenues to evaluate and motivate student learning and improve teaching that would probably be more productive. On a small classroom scale, for example, inviting subject area learning/teaching experts in to observe goings on might be an excellent way of getting initial feedback, at least. Side data concerning test results in science, math, and other technical disciplines might be somewhat informative but most likely not a panacea. You might query the "best teaching centers" (which might they be?!) to ascertain the extent they use statistics; but I'll bet they do not rely on them heavily.

In my opinion, the utility would depend on the quality of the data, the cost (time) needed and the inferences made thereafter. If you go by test scores, how do you know that these tests are valid and reliable enough for a particular type of analysis? (They seldom are).

Also, human learning is a highly complex endeavor, to say the least, which makes it utterly difficult to isolate even the major variables involved. Let's say you switch from some traditional teaching and learning activity to something that is considered novel or "innovative", and you observe some kind of significant change in outcome, it is really difficult to know what else could be at play here. Especially if the outcome it positive, we often tend to overreact and make too much of a single observation, in my experience.

That being said, I believe we do under utilize more advanced statistical methods when trying to understand student outcomes. We often just glance at averages and leave it at that.

To ensure safe results, there is a good combination of both qualitative and quantitative research.

Statistics help identify gaps. However, you have to define your scope and build rubrics to monitor and measure before and after the teaching and learning activity. Comparative analysis helps redirect or modify processes not achieving their objective to create the desired Learning Outcomes.

What about a randomised parallel group study? The condition for this would be that one group gets the "normal" teaching procedure (null hypothesis), the other (experimental) group gets the allegedly better teaching proceeding; the rmeasured results are to test by one-sided t-test or a similar statistical test procedure.

In nursing at the associate degree, baccalaureate, and graduate levels, we used standardized testing in specialty courses, sometimes at mid curriculum, and at the end of the program. We use regression analysis to determine the best predictive models. We also establish predictive accuracy for the exit exam with the licensing exam or the certification exams. We complete studies with new interventions using experimental and control groups. We cannot control for all of the extraneous variables, such as outside study and use of outside resources, but by using control groups, we may control for some.

http://irbis-nbuv.gov.ua/cgi-bin/irbis_nbuv/cgiirbis_64.exe?I21DBN=LINK&P21DBN=UJRN&Z21ID=&S21REF=10&S21CNR=20&S21STN=1&S21FMT=ASP_meta&C21COM=S&2_S21P03=FILA=&2_S21STR=VchdpuP_2015_130_76

There is a significant unmet demand on this area, Education industry is still lagged behind in using quantitative data to support education development.

Using statistic methods to observe learning and the effect of different teaching methods is plausible, but it has some major pitfalls. Like some have commented earlier, causality and correlation are not the same thing. When some group gets higher test scores, it doesn't necessarily depend on the teaching method.

- You might have made an easier test. If you use the same they might have learnt this and asked from others.

- You might want to improve on certain topic and discuss it more and the test has questions about this topic.

- etc...

Furthermore, changing teaching methods has usually a positive effect on learning because most times the students focus better during a new method. But it is not the medium what conveys the teaching but it is the message itself. You can have high-tech teaching style with poor content and end up with poor learning. Vice versa, you can have most boring and oldest method of just talking and not much else, but if the content is golden, the learning is going to happen.

Last but not least, as said by others also, the group comprises of individuals. Therefore the teaching and learning is heavily dependent on the group and what works for others might not work for others. I've noticed this with few years difference in undergraduate students: few years older were much more ready for open work and discussion and younger ones were better with conventional lecturing.

Ps. statistics also depend on the N. If you have just one class per semester and a small number of students, perhaps qualitative analysis would work better.

I think that most of the concerns voiced seem to revolve around getting a sufficiently large data set. This requires collaboration. But it has been done effectively. See Hake Plots of the Force Concept Inventory data.

Any study needs to be conscious of the statistical significance and limitations of the results, of course.

(R. R. Hake, “Interactive-engagement vs. traditional methods: A six-thousand- student survey of mechanics test data for introductory physics

courses,” Am. J. Phys. 66(1), 64–74 (1998).)

Dear researchers,

I really understand your point of view, even though there may not necessarily be a correlation between the learning / teaching process and the statistics. Obviously there are more fundamental aspects in that process, such as the content, the quality of the message and the prior knowledge that the students possess. The point is how to evaluate that the learning / teaching process is better? Statistical analyzes would be useful in these cases, how ?.

Best regards,

Ender

It should, also, be mentioned that in some analyses there may be an assumption that each student is the same as each other student and that they will all behave the same when exposed to stimuli.

It is always worth remembering that they are complex beings with complex behaviours.

I always worry about line/curve fitting to data where individuals are the source of information as a result.

Consequently, it may be that statistics can only be applied to those situations are easy to numerically quantify. I am not sure that it would be so easy to apply these methods to, say, instigating soft-skill behavioural changes in students (for example).

Following

It gives the teacher an analytical view to perform his various tasks, so he can treat weaknesses and consolidate strengths.

The processes associated with teaching analysis in the attached link can be used with statistical analysis, even if it is simple:

https://www.classtime.com/blog/teaching-analytics-students-learning/