Dear Angelo, Cronbach's Coefficient Alpha gives a measure of internal reliability which, in general terms of the meaning, is computed based on the mean scores of the inter-item correlations. Thus, the higher these correlations the higher the Coefficient Alpha. Furthermore, higher scores of Alpha indicate that all items in the scale perceived highly similar to each other by the respondents. In some cases, this can also be interpreted as  evidence for construct validity. On the other hand, in experimental studies, in contrast to cross-sectional design studies, there is always an intervention which may cause to change participants' perceptions regarding outcome variables (e.g., optimism, attitudes), and this, in turn, may reduce the inter-item correlations due to the possible effects of intervention on participants' perceptions. In fact, this is frequently observed in social psychology studies in which bogus negative/positive feedback is given to respondents. Therefore, I strongly suggest that you should consider how the intervention that you used in your study changed participants perceptions regarding outcome variable(s).  Furthermore, you should also investigate whether the changes of participants' perceptions in the group level may significantly contribute to variance of outcome variable(s). 

Altay Eren

Cronbach's alpha is an estimate of all possible split halves of the test. It is a measure of the repeatability of the results (without any intervention) if the test is given to the group of examinees again. In your case, it may be that the lack of knowledge for examinees in the pretest gave a relatively higher reliability value. This does not say anything about the posttest. In the post test, it may be that some examinees improved their knowledge base while another group did not, thus giving a lower overall reliability values. However, I believe in the post-test, you need to analyze the results by competency to see the effect of "learning" on a competency by competency basis. There may be something within certain competencies that influenced the inter-item correlations to a degree such that the overall  reliability value was lower. Regards

Basil Qaqish

Hello all,

Thanks, Angelo, for a real thought provoker.  

Altay and Basil are spot on the money.  

I created some faux numbers, kept the sum of pq the same, and solved for the variance of the test using your given Cronbach alphas of .77 and .44.  The difference in variance (for a 100 question exam) was 75 for the pre-test (assumes students not knowing the material and guessing at answers) and 32.5 for the post-test (assumes students learned something, as Basil did, above, so their guesses were less random and their answers varied less).

I made way too many inferences about your test, which was not my goal. I just wanted to run the numbers and solve this backwards to see if my theory made any sense.  I appreciate Basil's answer using split half reliabilities and figuring this out before me,  as well as Altay's very cogent discussion on the strengths and limitations of Cronbach's Alpha.

Cheers!

Rich

Would you suggest that I only report the reliability of the pre test and no longer report the reliability of the post test? I really appreciate all your inputs.

Another query:

Do the number of items affect the index of reliability?

Hi Angelo,

Yes it does, part of the formula is K/(k-1), where k is the number of items, so a 100 item test results in multiplying by 1.01, whereas a 10 item test results in multiplying by 1.11.  Also, the number of items will effects the total test variance, as the denominator is n-1.

Excellent question.  You can find the formula for Cronbach's Alpha here: 

http://hbanaszak.mjr.uw.edu.pl/TempTxt/Cronbach_1951_Coefficient%20alpha%20and%20the%20internal%20structure%20of%20tests.pdf

You can find the formula on Wikipedia, you will see it is not very complex...

Best wishes,

Rich

Hello Angelo

Please know that Cronbach's alpha is one measure or formula for reliability. The term "index of reliability" does not necessarily mean Alpha. The test analysis software "LERTAP" reports many measures in its analysis output. One of them is Alpha as we know it. But also it outputs an "index of reliability" value (usually slightly higher than alpha). Two different formulas are used in reporting the two output values.

The other thing I would like to mention is about the number of items. As a general rule the reliability value increases as the number of items increases. In a test, you are trying to "zoom in" on a latent trait. When you have more items, you do a better job zooming in, so to speak.

One thing I suggest is analyze your items using Factor analysis. Thank God for computers, SPSS does a great job in that. Just put all the items in SPSS and do a basic factor analysis (no need to get fancy with rotations, etc.). SPSS will output the "factor loadings" for you. They will be grouped together (in columns). That way you can see, kind of, which items may be "zooming in" on a  specific trait. If you have difficulty with this procedure and understanding it, you may need to speak to someone in the statistics department in your work place. I am sure they will help a lot. This has been my experience with the stat folks in general.

LERTAP does some basic factor analysis too. But not many people know LERTAP. This is why SPSS would be more suitable. A lot of people know SPSS all over the place.

Good luck Sir,

Basil

Hello

The post test reliability is bound to be lower than that of pre test because of the intervention. In the pre test the standard deviation will be high.  In the intervention the weak spots are strengthened so that all could perform well in the post test as indicated by high scores, lower standard deviation and lower Alpha value.  I think the reliability is tested only for pre test (will anyone who has done similar research come out with the way he/she did the calculation ?) If you want to do it for post test, the justification is 'effective intervention.'

I am answering with the logic of a layman.  Can statistical experts throw light on it?

Uma

Dear Angelo,

I also have seen such result in attitude scales. In your case, in the pre-test, test tool was measuring a quite similar thing, but, in post test, it did not. It means, either the respondents were not sincere, or, there was intervention effect so that perception of the respondents changed. 

Cronbach’s alpha is a measure of internal consistency, that is, how closely related a set of items are as a group.    It is considered to be a measure of scale reliability. A "high" value for alpha does not imply that the measure is unidimensional. If, in addition to measuring internal consistency, you wish to provide evidence that the scale in question is unidimensional, additional analyses can be performed. Exploratory factor analysis is one method of checking dimensionality. Technically speaking, Cronbach’s alpha is not a statistical test – it is a coefficient of reliability (or consistency). 

Cronbach’s alpha can be written as a function of the number of test items and the average inter-correlation among the items.  Attached, for conceptual purposes, we show the formula for the standardized Cronbach’s alpha:

Here N  is equal to the number of items, c-bar is the average inter-item covariance among the items and v-bar equals the average variance. 

One can see from this formula that if you increase the number of items, you increase Cronbach’s alpha.  Additionally, if the average inter-item correlation is low, alpha will be low.  As the average inter-item correlation increases, Cronbach’s alpha increases as well (holding the number of items constant).

The reliability of the test should be calculated by applying it to a sample other than the research sample.