If I understand your question correctly, the best approach would be developing an instrument measuring content mastery. This instrument should be based on the content of your class. To tease out the effects of technology on learning, an experiment with two conditions (technology: y/n) would be the standard methodology. Any difference in content mastery between conditions would be attributed to technology. However, one important note: It's not very likely that technology itself brings learning gains. Instead, the method that is supported through the technology is the active ingredient (see Clark/Kozma debate).

Let me know if this answer helps.

Dear Shirley Campbell,

I am writing to inform you that I read an opensource

book Instruments for Assessing Educator Progress in Technology Integration. It gives basic information about the instrument for assessing. https://iittl.unt.edu/sites/default/files/Instruments/InstrumentsforAssessingEducatorProgressinTechnologyIntegration_0.pdf

I hope I have been of some help.

Best

Andrija

I really appreciate all of your feedback! I am interested in the ways that we can show that technology is impacting learning, and I think that you have all shared something important. I agree with Andrija that it is important to measure learning with and without tech to see the differences in learning progress. It seems that the assessment of with and without is not as clear cut as I would hope. We often try to pinpoint the aspects of technology use that are different when students are learning, and ideas like motivation and engagement are discussed. Sometimes we look at the processes involved in learning and identify changes in the entire process because of the use of technology, like in the case of learning the writing process, where everything about the process is different from the paper/pencil version, including gathering ideas, outlining, editing, sharing and final publication.

So I guess I am looking for a list of instruments that can be used to assess any kind of change in learning with regard to technology. The book, Instruments for Assessing Educator Progress in Technology Integration, is helpful, as is the information at foundationcoalition.org. I'm looking for as many examples of ways to measure the impact of change related to technology use in teaching and learning. I've gathered some of the big measures that help with teacher and student beliefs and attitudes toward technology, but the measures used to pick up those individual instances of impact would be quite helpful as well, micro-level impacts, so to speak.

Thank you for the question There are many ways you can count on the calendar

Hi!

It is almost impossible to research learning with and without technology, since education itself is about technology in its core - writing to start with, print, video, etc etc - remember that not all technology is digital.

If you mean digital technology - watch first of all out for digital technology that basically does the same as earlier analogue technology, only faster, cheaper and more flexible. Not so much of a core difference. Does old kind of learning from a text on paper suddenly become e-learning when it is projected on a screen? Is it meaningful to measure any learning difference?

Next problem is that even if you make a reference teaching/leanring without any digital use - the students will anyway, uncontrolled by you, use digital ICTs to learn and communicate and get answer on questions and they will watch additional lecture videos on youtube if they did no understand something in class, etc.

You can make a big real Randomized Controlled Trial Study as in Medicine. That is very expensive and seldom done concerning teaching and learning. Take then only one, new digital tool you want to test any effects of on learning, run that in a group that uses the new tool, a group that uses some other digital technology and a reference group without - or similar - and compare them in a qualified way. Not so easy. Make it big so you compensate for individual differences concernign teachers and learners, etc.

In this light, for example the well-known case study of the type: Does learning increase if students blog online? and comparison last years results with this years result, can be interesting, but not of any meaningful evidence value. Almost all pedagogic / didactic research of this kind seems also always to be successful . but hard to replicate with other teachers and students.

I think personally that we can look for digital technology effects more successfully in the areas of access to education, inclusion, personalisation, possibility to study at own pace, etc.

This said, what you could have as help when elaborating a method, is the SAMR model for technology implementation in instructional design, see for example here: https://www.schoology.com/blog/samr-model-practical-guide-edtech-integration

Ask for example what technology substitution has been made, and how to test the value of that, what augmentation, modification, redefinition etc. You can find example of this kind of research I believe.

Seems that we are following similar lines of thinking. I have reviewed SAMR as well as ACOT, and several others that look at attitudes, beliefs, and levels of adoption and integration. I was actually looking for instruments that go beyond the kinds of pedagogic/didactic studies you mention above, and more for instruments that show learning gains using technology. I'm not really interested in comparing with/without, but instead, any integration that shows gains. Any tool that I find that is broad is also looking at the attitudes, beliefs, and levels of adoption and integration paradigm. I'd like to look at instruments that have already been created and used, if they exist. Thank you for helping me to clarify.

Thanks for the reaction. Yes I think these with/without technology comparisons are not so useful, and also very hard to make in any credible way that would show anything.

Another way is to make design-based research, DBR- partner with a teacher or teaching team as both designer and researcher (roles to be kept apart!) and try through a number of iterations to enhance their actual teaching process and measure how it results in learning. A problem is that this takes time; iterations can in regular schooling be done only once each semester or year and suddenly the course or teachers disappears. But it is a meaningful way of working.