Dear Leopold,

My advice is to use Principal Component Analysis first. Your 9 items might be loading on two different components. Have a look at my publication on this issue. The sample size > 300 requirement is also a bit folkloreish. What is more important is the size of the eigenvalue relating to the first component and the weightings of the items on this component.

We recommend carrying out a Cronbach's alpha analysis as a backup. The scale if item removed option in SPSS is useful here. The aim is to maximise the alpha whilst retaining as many items as possible.

Many thanks, Peter.

I am replicating a study that uses the Epistemic Belief Inventory (Schraw/Bendixen/Dunkle, 2002). It appears that this instrument has previously only been used with students. I am working with adults in continued education and I am having problems with items loading on different components than the model provides for - basically exactly what you are suggesting. Meanwhile I have read up on recent literature on Cronbach's alpha which confirm your recommendation. 

The trouble is that there is very poor correlation among items (partly below 0.3), although there is some clustering, which - as I understand - is a good sign. Prior to the PCA analysis I have checked KMO (poor at 0.670) and Bartlett's (0.000) measures.

I am using the German translation of the instrument and I fear that there are also content validity issues. So the head scratching goes on ..

You have a sampling problem, or rather a sub-sampling problem. When you select 100 cases out 0f 301 you can get somewhat different results, especially if you are selecting the 100 cases with replacement. In random selection it is always possible to select extreme cases. This can be selecting mostly outliers, or completely missing the outliers. Either way you get a sample that is more extreme than the population being sampled. This can happen in resampling, and it can happen in estimating the population values from a sample of 301.

@Timothy A. Ebert: this is a plausible explanation, thank you very much.

To get to the bottom of this: When I select my total sample of n=301 (alpha=0.687) on a specific variable, I get four sub-samples: 0.673 (n= 41), alpha: 0.646 (n= 121), alpha: 0.787 (n= 49), alpha: 0.682 (n= 90).

Is the reliability of my first sub-sample (0,673) as high as the total sample - or perhaps even higher - considering its size of n= 41? Or does the number of cases not (substantially) affect Cronbach's alpha?

Recent literature suggests alternatives   to alpha, and I have come across an interesting text by Lee Cronbach "My Current Thoughts on Coefficient Alpha and Successor Procedures" (undated, but probably from around 2000). In his concluding message he states that alpha "is now seen to cover only a small perspective of the range of measurement uses for which reliability information is needed.  The Alpha coefficient is now seen to fit within a much larger system of reliability analysis."

A smaller sample size will never be more reliable than a larger sample size. However, it is easier to find a small sample size that will support whatever ideas you have about the data.

You have a variable (say income: poor, low, middle, rich), that is used to select different subsets of the data. In this case there were poor (n=41), low (n=121), middle (n=49) rich (n=90). You calculate Cronbach's alpha for each case and find an unexpected outcome. I will argue that this is simply a random occurrence. If that is true then your observed alphas are nothing more than a random outcome out of all possible outcomes. So I would now take all 301 observations and randomize them. The first 41 observations will be group 1, Then next 121 observations will be group 2. Group 3 and 4 follow with the remaining 49 and 90 observations respectively. Record the alphas for each group, re-randomize the 301 observations and repeat. You will need at least 50,000 cases. Plot the distribution of alpha for each. How unusual are your observed values relative to the random outcome?

Got it, thank you. I shall report back.

Reporting back (and belated Happy Easter greetings):

Following Timothy Ebert's suggestion I randomized the sample (n=301) and extracted four subsamples (n=41; 121; 49; 90). I compared the alphas after doing so a few thousand times and found that the differences between 2,000 and 4,000 times were of in the order of 0.00x. The results after 4,000 randomizations were as follows:

n=41: alpha=0.674

n=121; alpha=0.684

n=49; alpha= 0.674

n=90; alpha=0.681.

Does this show that:

1. The differences in the alphas reflect the different subsample sizes

2. The originally observed alpha (0.787) for the subsample n=49 reflects the effect of that particular variable (and therefore not due to a random effect).

For the randomization I prepared an SPSS macro, making use of the OMS (output management system) utility. The code needs some cleaning, I will upload it when this is done.

Something is missing. After 4000 randomizations all of them returned an alpha of 0.674? I expected some variability in the outcome. If that is not present, then there is probably something wrong with the program.

Assuming that the numbers that you gave were means/medians, what you want is the distribution. So, out of 4000 randomizations a value of 0.787 or larger was observed how many times? You want to be able to say something like: We randomized the data 4000 times and a value of 0.787 or larger was observed 62 times.

A more detailed approach would look like this: We randomized the data 4000 times for each of 2000 sets. On average we observed 67.8 values greater than or equal to 0.787 and the 95% confidence interval to the p-value was 0.0562 to 0.00187. You can usually narrow this spread by increasing the number of randomizations.

Answer to 1) Yes, and no. It is more like an interaction between the discrete values in your data set and the sample size. Yes the variable you measured may be continuous, but in the data set that value is fixed/invariant. It may be 1.2578 but it will always be 1.2578 and therefore within your data set it is discrete even though its origin is from a continuous process. In that sense the answer is yes. However, if you mean that the sample size caused a particular alpha then the answer is no. If you go back to the field and collect a different sample of 301 and repeat this analysis then you will get different numbers. So a regression of alpha based on sample size would be inappropriate.

Answer to 2) You cannot answer the question based on the numbers provided, though you are closer to getting that answer.

Thank you for the swift reply.

Can we please focus on the n=49 subsample, which is my prime interest. Above I reported the mean only of 0.674, after 1000 randomizations.

11 (1.0%) out of 1000 observations were greater than 0.787. The 95% confidence interval is  0.670 to 0.678, p=.000. I am aware that I can reduce the confidence interval with more randomizations.

I understand your answer to 1), and that the sample size is not causal to the alpha value.

May I ask you what additional criteria are necessary for answer 2).

And yes, I do appreciate your time and patience.

The confidence interval is about the wrong statistic. So 11 out of 1000 observations were greater than 0.787. This is 1.1% but that value is not exact. There is a 95% confidence interval about this value of 1.1%.

Assuming that this works out and 11 out of 1000 is typical, then yes you can conclude that 0.787 is significant, and therefore the variable used to break these data into these groups is somehow important in your data set.

Now you have some harder questions.

1) If we look hard enough it is possible that we will see a significant outcome by chance alone even when there are no real effects to be observed. You looked at the other variables, is this the only significant effect you can find after sifting through all the possibilities? You could apply a Bonferroni type correction and divide 0.05 by the number of tests and use that as your alpha to identify significance. This is excessively conservative, but provides a different perspective. Is 0.011 still significant?

2) Does this observation make sense to you? Is it expected or contrary to existing theory (assuming there is an existing theory)? It happened in your data set, but do you expect it to happen again if you were to go back and gather another sample of 301 observations? Do you think this is repeatable?

3) Can you find data that would confirm/refute the validity of this outcome?

4) Realize that how badly you need this outcome to be true is inversely related to the probability that it is actually true. That is, we can transform and select specific outcomes and convince ourselves that we have found something of significance. The effort we apply to getting significance is directly proportional to the reward we hope to get. So if we are highly motivated then we will search endlessly through transformations and different methods to get that significant outcome, and stop at the first hint of success.

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

So, are other outcomes in these data consistent with this outcome, or is this the anomaly?

Thank you very much for your thought-provoking comments.

I have meanwhile repeated the above exercises (1000 randomizations) and arrived this time at 21 (2.1%) out of 1000 observations with an alpha greater/equal 0.787. As we speak, 3000 randomizations are taking place. It took me sometime to figure out to increase the iteration maximum in SPSS, which is set to 1000 per default. Also, on my home PC this is a task for several hours.

To you harder questions. It may be useful to provide some background  first. I am working on my doctoral dissertation and this particular issue is not key for my findings. Also, my supervisors will not in any way see my work negatively if I am not able to reject (any) null hypotheses, as long as I can demonstrate proper research is being done.

2) The construct in question is about beliefs in knowledge in the domain 'economics'. A sample statement to which respondents need to agree/disagree (scale 1-5) is: 'Truth in economics is a matter of opinion'. I am replicating a study, therefore I use the same instrument. My worry was that response may be influenced by what the respondent has in mind when he/she thinks about 'economics'. I have therefore added an additional variable, that is meant to capture exactly that. For this purpose I allow three categories (summarized): 1. accounting/costing 2. marketing/production/HR 3. micro-/macro theory, plus a fourth one for no answer.

There is a theory underlying these categories. Rules/facts in accounting/costing haven't changed over 100 years and longer, and are seen as rather fixed. For marketing etc. this is true to a lesser extent, and hardly at all for the micro-/macro theories. The latter forms the subsample of n=49 I am particularly interested in. If my new variable indeed has indeed an effect on response then it should be this group.

3) The file with 4x3000 cases got ready just now, I am now waiting for the analysis; because of memory constraints rather time consuming.

4) I am not desperate at all to prove what I mentioned under 2). Altogether I am a bit doubtful about the instrument and the underlying model. Research in epistemological belief is still in its infancy and there are competing models. There is general acceptance that such belief is domain specific; it is not clear however, whether a domain like 'economics' is granular enough.

I shall get back as soon as I got more results.

PS: 3000 randomizations result in 35 (1.2%) of 3000 observations greater / equal an alpha of 0.787

I am sorry that this takes so long. What I often do is to set up the program using 10 randomizations. As soon as I know the program will work, I wait for breaks. I can go to sleep, eat dinner, go to a movie, or go to exercise while the program runs. The results will be there when I get back. This gets more difficult with run times over 8 hours.

That was sort of useful, except that I am not an economist. "The construct in question is about beliefs in knowledge in the domain 'economics.' means almost nothing to me. I can define all the words but I don't get further. My most recent economics class is now some 30 years in the past.

Based on what you have done, it is reasonably safe to conclude that 0.787 is an unusual outcome.

In your four categories that you list in #2, it would seem that the people answering "micro-/macro-" are similar in some way. Is this an answer that is more typical of someone with a Ph.D. in micro-/macro- economics, while the other options are more characteristic of people with M.S, B.S. degrees or with fewer than 7 years of experience? These 49 people are all professors in economics from the local universities and you happened to catch them after their monthly economist's club meeting?

Can you try something like cluster analysis? Don't include this variable and see if these 49 people cluster, and who amongst the others are most similar to this group? Are these the 49 surveys that Joe did, and there was some unforeseen difference in Joe's technique relative to Bob and Sarah?

You have guided me very well, thank you again.

The economics aspect is not quite as sophisticated as you may assume. The target group comprises adult students pursuing a Bachelor degree in business studies. They are all working in various capacities and are of various ages.

I have requested and received permission from their educational institute to request their participation in an online survey. From approx. 3,500 students I selected 1,700 (more or less randomly), 301 filled in the questionnaire (uncompleted ones already deducted).

Previous studies were carried out mainly with students of commerce or in vocational training, i.e., none with actual working experience. The main goal of my dissertation to understand better how adult learners understand 'learning success' and successful learning transfers. This may sound trivial, but new concepts and theories of adult continued learning show that there is ample space for research.

I received a degree in educational science, after some decades of working as computer analyst / programmer - therefore the interest in adult learning.

I have uploaded a description of the above problem, together with code fragments for SPSS OMS (Output Management System) and SPSS macros.

http://preview.tinyurl.com/alphatest-en

I am getting close to where I cannot help. It has been some 15 or 20 years since I used SPSS, and I didn't use it much then. I did some BMDP and others, but mostly SAS. I have started a bit of R which I like and hate. I'll try looking at this again in a few days when I have more time.

Keep in mind that you can get unusual results by chance alone, and within the confines of a single study it is not usually possible to figure out if you are just seeing a random event that happens to achieve statistical significance, or if the effect is real.

Leopold,

The usual reason that larger sample sizes yield higher estimates of internal consistency reliability for total scores is that there is more heterogeneity within the sample.  More between-examinee variance translates to higher estimates of coefficient alpha, with all other factors held constant.  However, that "all other factors..."  phrase means no change in inter-item correlations.  With subsets of your sample, that may not be true.  So, your finding that subsets can yield higher estimates isn't particularly surprising.

In general, it's good to recall that reliability estimates are what they are: estimates, which might not apply to the next data set you collect.

Good luck with your work!