Hello Riselle,

You can do BOTH. You may run reliability for each sample separately and also for all 900 participants. Then you may compare all separate samples on reliability and if the value is above .700 then great. If however, one or two values are below .700 you may reflect on why it happened. Maybe there are sampling differences. However it is unlikely as 300 participants is a relatively big sample and students or university staff may share to high extent similar characteristics.

Cronbach's alpha is indicating whether all items included in the questionnaire pertain to and measure the same construct.

Also beware of very high Cronbach's alpha scores. Here is an article on that:

Shevlin, M., Miles, J. N. V., Davies, M. N. O., &Walker, S. (2000). Coefficient alpha: A useful indicator of

reliability? Personality and Individual Differences, 28, 229-237.

Kind regards,

Witold

Thank you. If that is so, I will use all data gathered from 900 people. I am designing an evaluation instrument to measure building performance. However,

1. If I am going to perform reliability test with all the 900, will the result of reliability vary? My point is, 300 from each university (A,B,C) will be my sample size. If i am going to use cronbach's aplha in one setting  using all 900 people, reliability will greatly be affected. since i'm going to perform reliability analysis on three study areas. All respondents have different experiences, characteristics, and environment (university).

2. Or should I try performing 3 reliability analysis? One reliability for A, one for B then one for C.

You see, my objective is to design, develop and validate an evaluation instrument that can be used to measure building performance. 

Riselle,

You mentioned you are going to use university samples to measure building performance... what does this mean building performance? Thanks for clarifying. I just wonder what is the rationale for using university sample (students presumably) to test building performance?

Witold

Yes. There were a lot of studies that i have read which stated that building performance has a strong integration with its occupants. Studies mainly focused on residential housings, dormitories, high-rise buildings etc. The researchers measured building performance by conducting a satisfaction survey to its occupants. However, only few have conducted on university buildings. So, I came up with developing a questionnaire and distribute these among students from three different universities.

Riselle, see if this example helps

Riselle,

Your statement here says it all:

My point is, 300 from each university (A,B,C) will be my sample size. If i am going to use cronbach's alpha in one setting using all 900 people, reliability will greatly be affected. since i'm going to perform reliability analysis on three study areas. All respondents have different experiences, characteristics, and environment (university)

Reliability describes a relationship between a measure and a sample, never a population.  As such, it cannot be carried over from study to study, from group to group.  If you have three different samples, you need to demonstrate reliability with each of those samples, as you note above.

Mark

if i will perform three different reliability analysis, does that mean that there is a high probability that i will create three different final questionnaires? each final questionnaire may have different items left. 

if i will perform three different reliability analysis, does that mean that there is a high probability that i will create three different final questionnaires? each final questionnaire may have different items left. 

The information below is from SPSS FAQ:

What does Cronbach's alpha mean?

Cronbach's alpha is a measure of internal consistency, that is, how closely related a set of items are as a group. A "high" value of alpha is often used (along with substantive arguments and possibly other statistical measures) as evidence that the items measure an underlying (or latent) construct. However, a high alpha does not imply that the measure is unidimensional. Technically speaking, Cronbach's alpha is not a statistical test - it is a coefficient of reliability (or consistency).

A reliability coefficient of .70 or higher is considered "acceptable" in most social science research situations.

 will be negative whenever twice the sum of the item covariances is negative. This can be stated even more simply by saying that  will be negative whenever the average covariance among the items is negative.

If one encounters a negative value for , implying a negative average covariance among items, the first thing that should be checked is to see whether data or item coding errors are responsible. A common problem of this type is that the scale consists of some items that are worded in opposite directions to alleviate response biases, and the researcher has forgotten to appropriately recode the reverse scored items, resulting in negative covariances where the actual covariances of interest are positive. Another possibility, most likely with small sample sizes and small numbers of items, is that while the true population covariances among items are positive, sampling error has produced a negative average covariance in a given sample of cases. Finally, it may simply be the case that the items do not truly have positive covariances, and therefore may not form a useful single scale because they are not measuring the same thing.

Please see also the attached document.