For example, can we measure the quality of our textbooks on a chapter-by-chapter (or paragraph-by-paragraph basis)?
Does the question involve the effectiveness of instructional material in regards to motivating learners and facilitating learning, or is it about quality from a publisher's/writer's position?
Dear James
In my opinion there are several methods to measure effectiveness depending on the learning goals of the instructional materials
Essential precondition, however, is first to establish the goals of the instructional materials. Rarely, however is it perfectly clear to teachers what the designers of materials are aiming at and often teachers adapt materials to personal preferences and convictions.
Finally, even when there are clear goals and the implementation of materials by the teachers is more or less conform the intentions of the designers, there still will be many factors which will influence the effect of the instructional materials on the learners in the classrooms
Personally I choose for Educational Design Research (see McKenny & Reeves, 2012) as a method to test the effects of instructional materials
Testing of instructional materials is certainly a hard task. As Harry said it would certainly be done against the objectives around which the materials in question were prepared i.e what learning objectives these materials are aimed at. Then one may design an experimental study to see how effectively those objectives are achieved through the use of instructional materials in question.
What increases the complexity of this measure, however, is that it might be impossible to attribute any deficiencies to their correct causes. For instance all or any of the factors like instructional materials used, the instructor, the environment, the learner's motivation or even the leaner's cognitive abilities can be responsible for under achievement. But yet carefully designed experiment or a series of them are the best bet and may help pin point the causes.
Is it the instructional material or the match between core concepts and how they are presented to students with diverse learning styles? "Textbooks" are created under the assumption that students using the text have an understood base of knowledge. The importance then lies in a carefully designed pre assessment that really does measure the core concepts of what is being presented.
One way to think about this question is to think like Elliott Eisner ... Sometimes the things we measure don't really count, and sometimes the things that really count can't be measured.
That said, on a large scale, if we 'really have the need' to measure, aren't we interested in learning outcomes that are the result of some form of instructional implementation?
Curriculum evaluation is a whole branch of research. Yes, you can measure the quality of educational materials or experiences by analyzing students ideas. You can collect students' ideas in clinical interviews, assessments or performance tasks. Exactly how you evaluate your instructional materials will depend on what you are trying to teach and your goals for students as well as their motivation. If you have questions, feel free to contact me. I have done quite a bit of curriculum evaluation experience.
I think the proper way to test the effectiveness of instructional material is through a pedagogical experiment. There are many techniques for this. In principle, applying a material to a group of students and to the other group, the material will not apply. Finally, we can make an assessment based on the contents of the material and thus the way you can measure the effects.
If possible, compare two classes, one using the material and the other using different material. I've done that because I was extremely dissatisfied with the Spanish textbooks available when they changed to a communicative approach several years ago. Our students couldn't cope with that ammount of grammar without explanations in the textbook but were expected to use the language. Grammar concepts of their own language are too different and they never had learned the concepts of Chinese grammar and sentence structure at school. First, I developed additional material to help them, but then I decided to write my own textbook, where different grammatical concepts are introduced in smaller steps, where information is given on how and when to use, for example, male and female forms of nouns and adjectives, etc. Before publishing, I tried out the material in one class and used the original in another class. I also used questionnaires to get studetns' opinions on parts where they needed more exercises or more information. Comparing the outcome in form of assessment tests between the two classes, I adjusted the material until I was satisfied with the outcome so that the material was ready for publication.
Editors have some protocols that involve different levels, one is the topic, then the order of the material, the language and the images, they use questionnaires, and sometimes they analyze the way people interact with it, it's pretty cool measure.
For the purposes of assessing textbook quality, I think we can measure quality by testing whether the educational outcomes are satisfied.
I have my educational outcomes listed at the beginning of each chapter of my textbook. This means that the quality of each chapter can be tested by before-after tests on those outcomes. If there is a problem with a chapter, then further investigations can be conducted to establish which sub-sections are causing trouble. Later one can look at paragraphs, exercises, glossaries, images, tables, poor ordering, weak sentences, style,grammar, spelling, typography.
This article here proposes a couple of measures (e.g. a boredom index) that are calculated from transactional log data of the LMS webservers: http://dx.doi.org/10.1504/IJTEL.2012.051544
Could some of us focus a bit on courses where an instructor does not follow a book or publisher guided criteria for constant reference and framing the course? This means a course created because instructors have been dissatisfied with textbooks or are not finding appropriate textbooks to develop a course around?
In such case, we will frame our answers in a way that focuses on course design as well as developing best measures for the effectiveness of our instructional material.
General acceptance of the text book / instructional material by the peer academicians, review groups and the student body are sufficient for the effectiveness of the material. The contents should also help students greatly to achieve the course objectives and outcomes. It is important that the presenter of the material should be well versed in teaching skills and overall course objectives. It is the teacher and the textbook / instructional material that will jointly contribute to effectiveness.
Thinking twice about it: I think the question is also about: how we can keep up engagement, engage the disengage, prevent frustration, etc.
We've just published this free iPad book, in which we have several articles relating to this (see the Introduction and see the Grand Challenges chapter): http://bit.ly/tel-advances
@Kevin Stoda, Answering your question, if that's the case, you need to ask to students and maybe other users to read and use your material, in my case I "tested" material during a couple of years to find all mistakes and misunderstandings possible before send to a publisher.
You must in this case think about some issues:
1) Actual information: how much often the information must be updated. Some topics move so fast, for example, neuroscience research, if you want to update frequently your information, the best if using blogs or magazines kind of flip board, but if you are using classical information then probably you only must be sure is correct.
2) Your platform: what kind of platform you use? are you in digital? are you on paper?, maybe you have your information in power point, no matter what you are using, be sure you can read it in all devices possible. Think about your students, most of them have intelligent phones? well, they read information when they have a chance, in the train, bathroom, before sleep, when they wake up. If you give them the chance to have around your material ALL THE TIME, you can be sure they will read it.
3) Your students habits: why do they read? they read because they HAVE to? just before the exam? or because they enjoy reading?. My experience in Mexico tells me students read because they have to and if they can avoid it they will, so tell them catching stories a blog would be an option, keep your information fresh and the good thing is you can add an option to readers tell you about the quality, so you can know for sure your readers opinions.
4) The kind of information: is it a very particular topic or is the kind of information you can use with other topics, something called horizontal curriculum, this means your information can be used to different subjects, this helps so much because students can have it around. If you prepare super specific information, first of all, you will use it just now and then and secondly, it will be hard to find.
5) Ask "peer review", let your colleagues to give their point of view. It's always good to have a fresh reader.
Even if the learning outcomes are correctly specified, and are correctly met by the educational material, how will we measure the effectiveness of it over the lifetime career of the student?
Using active learning ,learning by practices ,applying,case studies to simulate the job market environment
@Ian Kennedy, that would be a huge research following students after our classrooms, but that means we believe there is A learning and neurocognitive research shows that doesn't exists in real world. I am pretty sure nobody teach how to design an Ipad or how to steal a bank using a computer, brain takes the knowledge creating skills and those skill can be transformed on new skills, new paths of answer attending the environment.
Learning is not static, brain is not static, real life is not static.
I guess one important dimension if the material matches in content beside the educational values the cultural values. I used a quantitative methods book for years from several western publishers, too many examples are there that students do not read or care to read. So, we published a book with examples from the reality of lives of the students without loosing focus on the technical knowledge. It has been very exciting to see students reading from cover to cover and comment about the examples.
Perhaps looking at the problem a bit differently, there are many different issues here. When do you a book is effective? Here are three answers. 1. The book fascinates the learner to spend more time on the book, to read more and more. 2. The book helps the learner to attain the competencies expected in the assessment of the subject (e.g. final examination, project work, whatever), 3. The knowledge acquired turns out to be valuable in the long run. All three are important, and need to be looked at. Of course, 1 is perhaps easier to measure, and 3 the hardest.
If the resource is online and accessed online, a bunch of analytics that can be gathered from the portal may provide some clues to answering the various aspects of the question.
If the material sufficiently self guide the students on the subject discussed, then it will be a good instructional material
Students often struggle with books and the academic language, but mastering the material is also part of their learning.
I think that unless you are referring to self paced, learning, that the instructional material can not be evaluated without reference to the tutor. The role of the tutor is to translate or create a bridge that enables the student to grasp the concepts. Remembering that knowledge is acquired and is understood as a multi layered experience and is not always a linear process.
I have many books that I studied in college and found difficult at the time, but now years later I still refer to them when writing courses, etc. They are still effective as resources.
Jane Leonard, the books' being effective for an expert does not imply that they are effective for a student as well. Particularly, as you have mentioned yourself, the fact that the books were difficult for the leaner they were meant for, itself implies that these instructional materials were not effective.
To measure the effectiveness of our instructional material, for example on a chapter-to-chapter basis to ascertain the quality of a textbook would tantamount to taking a very simplistic view of the whole process and undermining the real essence of the evaluation exercise. There are numerous readily available instruments to assess the effectiveness of instructional materials. And this exercise should also take into account several aspects for which the whole measurement or effectiveness study is being carried out, i.e.: the lesson/course objectives and outcomes, the context and learner analysis, other practical issues and concerns, etc. For example, if learners' motivation is of concern to the researcher/teacher, then John M. Keller's IMMS (Instructional Material Motivational Survey, 1993) could be used to a great effect by all means, and so on.
This is one of the most difficult topics to do good research on. That is because the use of instructional materials cannot be separated from many other factors that could affect their effectiveness. The most common way to test the effectiveness of teaching materials is to compare two classes; one using the set of materials that you want tested and another using another set of instructional materlals. One class serves as control. However there is still a problem: there are other things that affect the effectiveness of the materials that cannot be controlled well -- there is teacher personality for one. One teacher could implement certain materials better than other materials. One solution people take to overcome this problem is to use the same teacher for both experimental and control class. This teacher should be one presumably one who is not aware of the purpose of the research. This person can teach one class using one set of instructional materials then he or she teaches the other class using another set. Questions still arise. Will this person teach the same content using two instructional methods or different contents with different instructional methods? That does not seem natural and variations can still exist. The second time you teach something is never the same as the first time you do it.
OK, she teaches one class first using instructional materials 1 and the term after she teaches another class using another set of instructional materials. Bu then there is the matter of the students, their motivation, etc that can also affect the results. Even if all things are equal in terms of student motivations, teacher personality etc,, there is still the question on whether the application of the instructional materials today and in this place is the same as it application in another day and place. The interaction between teacher and students and materials can differ from time to time. I am a very experienced teacher and I know my stuff very well but there are some days and with certain classes when my teaching is more superior Research on task performance designed to elicit specific targeted elements (e.g. the simple past) suggests that the number of incidence of targeted elements will differ according to many conditions. In other words, to really make sure that the gains attributed to the use of instructional materials is due to these materials and not to something else is really difficult to do. Does it mean that materials cannot be ever compared as to their effectiveness? I am not prepared to accept that. I am sure they can be. It just requires a lot of caution. I am glad you have posted this question here so different people can contribute different ideas. Maybe a few of us can get together and discuss how this comparison be done because it is a necessary thing to do.
Great question! Looking forward to the answers, which I hope will include input from K-12 teachers.
Read Elliot Eisner's 1971 article. "How do you measure a rainbow? Tactics for evaluating the teaching of art" from Art Education, 24(5), 36-39
The instructional material that holds attention span of learners and that meet the need of learning styles of most learners.
Evaluation ca take any shape either formative or Summative, which is to analyze the students during the study and at the end of the course. Now it depends on the requirement of time can measure chapter by chapter or lecture by lecture. Course or curriculum evaluation can also be done by following some evaluation model like Tyler model.
- Badges
- Science topic