Test validity can only be assessed in the context of what the test is being used for. Are you trying to measure something that you would not expect to change over the course of the test administration? If so, are you sure that you haven't somehow ended up with easier items at the end of the test (if your items vary in difficulty level)? If you're trying to measure something that you don't expect to change, and you don't have the easy items at the end, then you may have a problem for validity.

Thank you, Victor!

Actually, I did expect it to change. Befor analysing the data, I expected either a decline on performance, due to tiredness, or an increase, due to the fact that individuals could learn and get better throughout the test. My doubt, then, is if this affects the way in which I should analyse the data.

As a example, should I not consider the same weight for the first and the last items on the test? Once that an error could be caused by 'inexperience' on the first items, indviduals should not be as much penalized as they should if the error occurred on the last items.

I don't know if I made myself clear, but those are the doubts I'm dealing with right now.

thanks,

Pedro de Alcantarap

Hi Pedro,

Is this test your dependent variable? If the main question in your study is whether there's a change in performance over time, then it doesn't seem like you have a validity problem, you just have the result. For example if your hypothesis was that learning would increase scores or fatigue would decrease scores, then you can analyze the trend and you have your answer.

If the test is being used as a predictor, or if it's an outcome variable and you're *not* interested in change over the course of the test, then you may have a validity problem, because it seems that performance on (part of) your test is capturing more than one construct. You haven't said what the test is or what it's supposed to measure, but just to take an example from a simple test: Researchers using reaction time tasks usually give participants quite a few practice trials (or they just don't look at some number of trials at the beginning) so that performance can stabilize. This is because people are getting used the task, equipment, etc. and researchers don't want to measure that, they want to measure how quickly a person can respond to the stimulus, once everyone is comfortable with the task. So, if there's a chance that the increase that you have observed on your test is happening because people are still figuring out how to respond to the test items in the beginning, then you may be capturing variance that is not what you're trying to measure.

I'm not familiar with weighting responses differently depending on where they occur in the test. I suppose it could be done in theory, but I think you would need a very good justification for it unless it is commonly done with this type of test in your field.