I understand that you were asked to evaluate a textbook.

You have heard from your reference that validity of research can be analysed under the three headings: content, criterion, and construct validities. So I take it that the textbook is on Social Science research.

I presume further, that you were not given a rubric to evaluate the textbook, so you wish to invent your own. In the bigger picture, you should be looking at:

Your views of the above will differ from those of experts in the field, those who need to lecture from the text, and the students. Within these groups, there will be a different rubrics and differences of opinion. This is natural and normal.

I understand that you were asked to evaluate a textbook.

You have heard from your reference that validity of research can be analysed under the three headings: content, criterion, and construct validities. So I take it that the textbook is on Social Science research.

I presume further, that you were not given a rubric to evaluate the textbook, so you wish to invent your own. In the bigger picture, you should be looking at:

Your views of the above will differ from those of experts in the field, those who need to lecture from the text, and the students. Within these groups, there will be a different rubrics and differences of opinion. This is natural and normal.

Approach

Dissemination of a vision (theory or approach) about

Content Presentation

• Stating purpose(s) and objective(s)

• Selection and its rationale

• To the teacher

• Supplementary materials
• To the student
• Piecemeal, unit-by-unit instruction
• Graphics (relevant, free from unnecessary details, colorful, etc.)
• Periodic revisions
• Workbook
• Exercise and activities
• In the classroom
• Homework
• Sample exercises with clear instructions
• Varied and copious
• Periodic test sections
• Accompanying audio-visual aids

Physical Make-up

http://iteslj.org/Articles/Ansary-Textbooks/

Thanks a lot for your support. I especialle liked the Paper of yours, Krishnan!

Nevertheless my question has a little different meaning: According to some aspects of validity (content, substancial, structural, external etc. see Messick, 1995), I'd like to know which of these validity-aspects (NOT content-aspects) I adress by comparing expert ratings of a textbook with student rating...content-validity?

Hi Daniel,

that depends on what you actually asked the students and experts. Expert ratings are a means to evaluate various aspects of validity, among them content validity. Did you ask them e.g. to rate whether or not all contents of school science are covered in the books or if the content of a test inventory is represented in a book? If you did so, you might have investigated content validity. Did you ask them to rate the quality of textbooks? If you did so, this could be an aspect of criterion validity (or convergent validity, it depends on the terminology you use). But it is only criterion validity if you are sure that you measured the same with different methods. To understand the differences in their answers you have to get aware of their different understandings of quality (or whatever you asked them). Maybe the students value other aspects than your experts (teachers?). To understand those differences you have to conduct interviews. However, if you think that students and experts answer differently because they actually evaluated different things, you did not research validity at all - you researched the differences in students' and teachers' judgments. That is because you measured different things with different methods in that case

Consider reframing as cultural politics:

Apple, M.W. & Christian-Smith, L.K. (1991). The politics of the textbook. In M.

W. Apple & L. K. Christian-Smith, The politics of the textbook. London, UK:

Routledge.

Apple, M.W. (1992). The text and cultural politics. Educational Researcher,

21(7), 4-11, 19.

Apple,M.W.(1993).The politics of official knowledge:Does anational curriculum

make sense? Teachers College Record, 95(2), 222-241.

Also, I have done work on evaluating the PROCESS of textbook creation - which I think adds an important dimension. Through the reductive process of editing, authors may not be so happy about the final product, either :)

@Christoph: "Did you ask them e.g. to rate whether or not all contents of school science are covered in the books or if the content of a test inventory is represented in a book? If you did so, you might have investigated content validity."

=> Yes, I did as a first step and knew I was investigating on content validity

"Did you ask them to rate the quality of textbooks?"

=> Yes, that was my second step and core of my question here on researchgate: Can I argue, that different groups (experts vs. students) make a different method, like you wrote? They rated on the same book, so the "things" to measure are equal. Then, this would be an easy convergent validity study...

@ Daniel: well, that depends. They might have to rate the same book but still have very different quality criteria in mind. I am not very much familiar with the research on textbooks, but e.g. for explanatory videos on youtube it is known that students sometimes think highly of videos that are of a poor explaining quality - just because the videos are well-animated or whatever. It could be similar with textbooks.

In my opinion, it would be a study on convergent validity if you can argue convincingly that the perspectives of the students and of the experts on quality have some kind of common ground - only in this case you measured the same (textbook quality) with two different methods. Both groups have to have a shared picture of quality - otherwise maybe the experts judge based on the color and the students based on the number of words, comparing those results would be of little use. If you have a "model of textbook quality" and you can assort those aspects the students value and those the experts value to this model - than you measure quality by using two different methods and then you might argue that the results support convergent validity.

Just a remark: why do you need an argument for convergent validity after all? I suppose you did not ask the experts for high inferent judgments but asked them a number of questions with respect to various aspects of textbook quality. If these aspects are backed up by a well-developed theoretical model and the interrater-reliability is appropriate I do not see the need. You might consider using your experts as one group and the students as a different group of experts for textbook quality and what you might have done was the start of a delphi study to find better criteria for textbook quality. But that all depends on what the goal of your research is...

I hope I could help!

@Laura and @Krishnan: Thanks for the "Theory", I'll build it into my paper :)