If your survey does not adopt a theoretical framework with constructs to validate or hypotheses to prove, then internal validity is less of a concern. However, external validity should be always be ensured whether your survey research is descriptive, relational or predictive; whether your instrument is long or short. Not a good idea to abandon probability sampling.

Hello Alexis,

First up, in my response below I am going to refer to validity in a different way from the way that Alexander has above. He is, I think, drawing the useful distinction between internal and external validity in the context of whether you can be assured of causal links in your investigation (internal validity) and whether your results will apply more broadly than your investigation (external validity - or generalizability).

I am approaching your question from a different angle, i.e., the processes involved in creating a satisfactory measuring instrument.

My view is that, no matter how short or long an instrument is, it is ineffective if it is neither reliable nor valid.  Therefore, even if you are not able (or inclined) to obtain a lot of evidence that your instrument is reliable and valid, I think you should make SOME attempt to do so. This might require you doing some work prior to your main investigation.

At a simple level, for reliability this might involve some pilot testing to investigate test-retest reliability (consistency across time) and item-total correlations or Cronbach's alpha values (for looking at similarity among your items). It might be too much for you to use things like factor analysis as a means of exploring internal consistency.

As far as validity is concerned, you might have (or want) to rely on only some basic forms such as face and content validity, but there are other forms of validity that you might be able to employ.  In fact, there are at least 11 forms of validity in addition to face and content validity. Here is an article that I wrote some years ago in which I describe different forms of reliability and validity, as well as some other features of measuring instruments.

Trevethan, R. (2009). Self-assessment of foot health: Requirements, issues, practicalities, and challenges. Journal of the American Podiatric Medical Association, 99, 460–471.

Don't worry that this article has a podiatric focus in places. The points I make are readily applied to other contexts  And please be aware that at one point I made an incorrect statement about intraclass correlation coefficients (ICCs). I said that they were essentially no different from standard correlation coefficients. That is not always true. Sometimes it is, sometimes it isn't - depending on the type of ICC that is used. This might not matter to you, however.

Every good wish with your research.  If I can be of further help, please feel free to get back.

Robert.

Dear Alixes 

Every scale has been constructed and administered on their own cultural settings, therefore, the way of receiving and processing the information is different from the other learners. I think the best way is  is check again the reliability and validity of the scale which you want to use. 

Mehraj