Dear Razeen,

Probably you can look at this post https://www.researchgate.net/post/Do_you_know_any_rating_scale_evaluating_motivation_for_learning

where people posted some questionnaires which used to assess various aspects of motivation (goal orientation, self-efficacy). depends on how you define "engagement", the questionnaire on self-regulation may also be useful for you to explore how engaged students are (i.e., how well self-regulated skills students can deploy for their learning).

Dear Razeen,

despite the fact some researchers tried to propose a scale to measure the student engagement, I think that it depends too much from the specific objective of the tool/resource you are analyzing.

Maybe also the (socio-psychological) concept of "self-efficacy" can be useful for your analysis, if you think the result of an improvement in your material will provoke an improvement in one or more skill.

I saw it used for validating the impact of ICT adoption in K-12 teachers in the context of under-developed countries...

Based on my experiences, I have never seen a validated scale, but I noticed that every research group adopts its own (even thought sometimes they differ just by little details).

Good question, Razeen. I would also be interested in what other researchers are doing in this space. The notion of a "validated instrument" for motivation sounds challenging, though, & e-Learning has become quite multi-disciplinary. We are all motivated by many things in life & understanding why we do things can sometimes demand intensive reflection. Motivation can have many facets and can often be explained in terms of "reasons why". One facet is "aspiration" & a project I am involved in at the moment is looking into this with a targeted cohort of students over a 2-year period. As for engagement -- while there may well be a correlation between motivation & engagement perhaps there are situations where motivation has nothing to do with it? For example, an impulsive (engaged) interaction with a compelling online resource (that may have been explicitly produced as a learning resource or not).

Engagement could just be the time spent with various parts of your e-learning materials, which you could measure. In a Web environment this leads to a typical Web analytics application where you, e.g., try to improve the effectiveness of your Web site. Offering two versions for comparison could be a starting point.

i can send you the 62 point scale for measuring motivation and engagement of students with e-learning materials. You can mail me at [email protected]

This is a good question, among other things, because it generates new questions, like: what is motivation/engagement? In order to measure, one needs to know first what is measuring.

Perhaps this publication will help: Kim, K. J. (2005). Adult Learners' Motivation in Self-directed E-learning, Dissertation. Bloomington IN: Indiana University (Accessed 12/21/2012 http://books.google.com/books?id=Z0RJMwAACAAJ).

Following is a relevant questionnare:

E-learning Attitude Scale

1. E-learning does not scare me at all

2. Working with a e-learning would make me very nervous

3. I do not feel threatened when others talk about e-learning

4. I feel aggressive and hostile toward e-learning

5. It wouldn’t bother me at all to take e-learning courses

6. E-learning make me feel uncomfortable

7. I would feel at ease in an e-learning class

8. I get a sinking feeling when I think of trying to use e-learning

9. I would feel comfortable working with e-learning

10. E-learning make me feel uneasy and confused

11. I’m no good with e-learning

12. Generally I would feel OK about trying a new problem with e-learning

13. I don’t think I would do advanced e-learning

14. I am sure I could learn through e-learning

15. I’m not the type to do well with e-learning

16. I am sure I could learn how to handle e-learning

17. I think using e-learning would be very hard for me

18. I could get good grades in e-learning courses

19. I do not think I could handle an e-learning course

20. I have a lot of self-confidence when it comes to studying through e- learning

21. I would like to study via e-learning

22. The challenge of solving problems with e-learning does not appeal to me

23. I think studying through e-learning would be enjoyable and stimulating

24. Figuring out e-learning problems does not appeal to me

25. When there is a problem with e-learning that I can’t immediately solve, I would stick with it until I have the answer

26. I don’t understand how some people can spend so much time studying through e-learning and seem to enjoy it

27. Once I start studying with e-learning, I would find it hard to stop

28. I will do as little study with e-learning as possible

29. If a problem is left unsolved in an e-learning class, I would continue to think about it afterward

30. I do not enjoy talking with others about e-learning

The questionnaire is in an advanced stage of development by Yaacov J Katz and Yaacov B Yablon and has been piloted a number of times in the Hebrew version only. In the last pilot study (2012) the questionnaire was administerd to a sample of 132 first year social science students at an accredited Israeli university. Validity was achieved by evaluation of the original 48 items by a panel of 5 e-learning experts well-versed in the use of attitudinal questionnaires (who recommended that 18 items be discarded) and reliability for the 30 item questionnaire (Cronbach Alpha) was computed at the .87 level.

Dear Yaacov, If I use some of the items, how can cite this in my work?

Thanks

I suggest that you cite the work as follows:

Katz, Y.J. & Yablon, Y.B. (2012). A reliability study of the e-Learning Attitude Scale. Work in Progress, School of Education, Bar-Ilan University.

Dear Razeen

during my PhD study I developed an instrument which measured motivation to learn along with e-learning usability. Actually that was the main outcome of my doctoral thesis: the development of a new questionnaire-based usability evaluation method that proposes motivation to learn as a new type of usability measurement for e-learning courses. Two large empirical studies were conducted in authentic business environments, with real learners-employees in corporate settings who took several e-learning courses and evaluated them by using my instrument. Results provided significant evidence for reliability and validity of the instrument. This work was completed in 2004. Since then many scholars have used my theoretical framework and instrument.

You can find more details about the development and the validity of this questionnaire in the following publications:

Zaharias, P. and Poulymenakou, A. (2009). Developing a usability evaluation method for e-learning applications: beyond functional usability. International Journal of Human-Computer Interaction Vol 25 (1), January 2009, 75 – 98.

Zaharias, P. (2006). A Usability Evaluation Method for e-learning: Focus on Motivation to Learn. In Extended Abstracts of Conference on Human Factors in Computing Systems – CHI 2006:

Fyi, I am attaching a version of the instrument.

Best regards

Panagiotis

Thank you Panagiotis, and also to everyone else who has responded. I really do appreciate all the very useful contributions.

HI I have done similar work but not completed yet.. You can have a look at this paper which is the first draft of the work.. we have a system to evaluate the involvment of the students in online discussion.. I used Multimedia as well

I think the following can indicate motivation and engagement of the student:

1.The number of times the student comes to the e-learning class.

2.The number of times and number of weeks the student participate in the e-earning forums.

3.The numbers of times the student vote on a post reply or start new thread.

4.The number of times the student active in the forums.

5.The number of times the student read thread from classmates.

6.The number of times the student vote on interesting or useful posts.

7.The number of times the student post good replies (e.g. earn 3 likes or more).

8.The number of times the student start good threads.

9.The number of times the student write a post that classmates vote up a lot.

10.The number of times the student start a thread that classmates vote up a lot.

11.The number of the times the student looks at the online quizzes and solves them.

12.The number of times the student do the e-learning course assignments

You may like this piece (appeared in JASIST):

Heather L. O’Brien and Elaine G. Toms (2010): The Development and Evaluation of a Survey to Measure User Engagement, In: JASIST, 61(1):50–69.

https://www.researchgate.net/publication/220433691_The_development_and_evaluation_of_a_survey_to_measure_user_engagement?ev=srch_pub

Article The Development and Evaluation of a Survey to Measure User Engagement

the Gallup Organization Q12 study offers an instrument for measuring worker's engagement. Wonder if you can apply it to your study.

Thackray, J. (2001, March 15). Feedback for real. Gallup Business Journal. Retrieved from http://businessjournal.gallup.com/content/811/Feedback-Real.aspx

Medlin, B., & Faulk, L. (2001). The relationship between optimism and engagement: The impact on student performance. Research in Higher Education Journal, 13, 1-9. Retrieved from http://search.proquest.com.library.capella.edu/docview/889136489?accountid=27965

Does anyone have this journal or know how to get it

The title is: "The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education"

It is available online:

http://www.selfdeterminationtheory.org/SDT/documents/1992_VallerandPelletierBlaisBriere_EPM.pdf

Thanks!

Hi Razeen,

The following link contains some useful relevant instruments that may be of help for you :http://itestlrc.edc.org/STEM_education_instruments?page=3 and www.inn.colorado.edu.

Thank you Ali, looks like a very useful site.

Thank you Kathy, this is really useful!

Allow self directed participation and track the depth and/or how far into the program your users delve into or participate in. If you find that your users only interact with the first few layers of your program, chances are they weren't motivated to interact with your program. Then ask your users why?

Another method is to test retention of the material. People who are interested in material tend to be motivated into learning that material. Of course, in your falsification, don't forget to account for any and all of the various learning disabilities, and different learning styles. You might have users that are motivated to learn, but are unable to learn within the parameters of your e-learning program's delivery system.

Useful papers :

1.Modeling the Assessment of Quality Online Course: An empirical Investigation of Key Factors Affecting

Learner’s Satisfaction, Neila Rjaibi, Latifa Ben Arfa Rabai, IEEE TECHNOLOGY AND ENGINEERING EDUCATION (ITEE). Vol 7, No 1 (2012), edited 23 March 2012, ISSN 1558-7908.

2 Assessing Quality in E-learning including learner with Special Needs, Latifa Ben Arfa Rabai, Neila Rjaibi, Proceedings of The Fourth National Symposium on Informatics, Technologies for Special Needs, April 23-25, 2013. King Saud University, Riyadh, Saudi Arabia

The first paper

Thank you, Michelle and Neila, for your suggestions.

Regards

Razeen

Thanks, everyone. I have been trying to get O'Brien el al instrument but in all articles I read the instrument itself is not attached.. does anyone have it? I have written to the authors but no luck:-(

Hi- we have developed a detector that detects students' engagement /disengagement in on0line enviornnments. We have a few papers on this. Would you like them?

Janice Gobert

[email protected]

Hi Janice, that would be very useful so please let me have a look at your papers.

Regards

Razeen

[email protected]

Article DESIGNING AN INTERACTIVE EDUCATIONAL WEBSITE USING ASP.NET.

E-portfolio best tools to do that

Does anyone have an instrument that can be used with grades K-12? Some of the language of these suggested motivational measures seems very high-level for a grade K-5 student.

I am also working on something related to this in my ph.d. I will send to you as soon as I have a developed instrument. If you find too, kindly let me know. Thanks.