I'm intrigued. Why do you want to establish  (just) face validity, which is simply the perception that a research instrument (say, a questionnaire) measures what it is supposed or intended to measure (is valid); but leaving things at this can be misleading as your research instrument may not, after all, measure what it is intended to.

However, validating that your research instrument is actually measuring what it is supposed to is what  instrument piloting is for, and this seems to be what you are doing.

And in regard to piloting, what you really want to know is  whether your questionnaire is understood and answered  by respondents much as you intend.

How many respondents do you need in the pilot? This is really a judgement-call in light of your intended research; however, for robust piloting, I don't think that you'd go wrong with say, 5% of your intended  actual research sample.

Hello Bethany,

Like Philip above, I would begin by asking something along the lines of "Why (only) face validity? Maybe, however, at the moment you're concerned solely with face validity and you'll deal with other forms of validity separately - so I'll begin by focusing on face validity.

In some respects, face validity is one of the weakest, most superficial, forms of validity. Essentially, it is simply concerned with whether an instrument LOOKS as if it will do the job.  Having face validity doesn't guarantee for a minute that an instrument really DOES do the job.  That said, face validity can be important for establishing and maintaining rapport with the people who use an instrument - either as administrators or participants.

In order to establish face validity, my sense is that you don't really need to conduct the kind of pilot study that involves people using/completing your instrument, but rather asking a number of informed people whether they think that your instrument looks as if it will do the job.  I'd begin with people who are, for a variety of reasons, "in the know".  Perhaps when you feel that you are getting to saturation point - i.e., all of those people are giving you the same kind of response - you could feel safe about face validity. In the process, you might adjust the instruments that you started with.

Face validity is reasonably allied to content validity, though they are a bit different. Forgive me if I am moving too far away from your original question, but my hunch is that it is important (for most instruments) to ensure that content validity has been attained, and then to look at face validity.

In case it helps, you might like to read an article that I wrote about developing instruments to assess foot health, but it applies to instrument development in general. Here's the reference:

Trevethan, R. (2009). Self-assessment of foot health:  Requirements, issues, practicalities, and challenges.  Journal of the American Podiatric Medical Association, 99, 460–471. http://dx.doi.org/10.7547/0990460

I am a bit ashamed because I wrote some inaccurate things about intraclass correlation coefficients on page 464, column b - so please disregard what I said about them always being equivalent to correlation coefficients. That's true only under some circumstances.  Apart from that, I think I've covered quite a lot of territory about instrument development in that article, including validity and reliability as well as some other topics.  Take it or leave it!

Please feel free to get back if I might be able to help further.

All the best with your research.

Robert.