The first step is make sure that all the items are in the same direction, by dealing with the reverse coding. I assume that two items with "perfect" scoring means that everyone gave the same answer, so you will have to delete them because they have no variance, and thus cannot correlated with any thing.

After that, run Cronbach's alpha using the "alpha if item deleted option" which will tell you if your reliability would go up after removing any given variable from the full set.

Run a Factor Analysis and look at the loading. First eliminate low factor loading questions and the look for cross loading ones. That might help you.

The first step is make sure that all the items are in the same direction, by dealing with the reverse coding. I assume that two items with "perfect" scoring means that everyone gave the same answer, so you will have to delete them because they have no variance, and thus cannot correlated with any thing.

After that, run Cronbach's alpha using the "alpha if item deleted option" which will tell you if your reliability would go up after removing any given variable from the full set.

Hello Rani, 

Exploratory Factor Analysis would be an appropriate "tool" to deal with the issue you are presenting. You could look at standardized factor loadings then and analyze them. It would be also appropriate to repeat your study using different samples. Also you could perhaps build some subscales or group questions pertaining to some specific areas of literacy. Perhaps if you have many questions on one of the areas you could reduce their number...

Best wishes

Since you can’t do a CFA, and you have nominal measures I don’t know of a statistical method to use. However, I remember using expert judges to rate the degree items fit a construct on a scale development paper I worked on.  Have experts rate the items on the degree they measure the desired construct and use the highest rate measures.

Thank you all for your response.

@ David Morgan, when I remove the items with no variation, the Cronbach's alpha score is not robust. Any suggestions? Should I keep aside all the items with no variability and not use them at all?

@Wilkinson, please send me the paper you have cited. Appreciate it very much.

@ Joe, I am thinking of using Kurtosis (to see the spread of the data) and accept only those items whose ranges fall in the leptokurotic range.Any thoughts about that?

I’ve never seen it done that way. My concern is that it would be viewed as item reduction through data fitting by reviewers.  Due to the variation issues I think you have to have significant conceptual backing or there will be trouble during review.

I'm not sure what you mean when you say the Cronbach's alpha is "not robust" if you remove the items that have no variation. Do you mean that alpha drops to a low value? this could mean that the correlations among these items are calculated as 1.0, so dropping them would indeed reduce your alpha -- even though those high correlations are essentially meaningless.

I suggest that you haven't already done so, that you spend some time examining the original correlation matrix.

Also, these measures are ordinal, not nominal, and your goal is to create a single interval scale where all the items are theoretically specified as indicators of the same concept. That is a straightforward approach, but you do need to remove items that don't meet those assumptions.

Hi David:

How can yes, no, don’t know responses be ordinal?

Yes = 2

DK = 1

No = 0

Three items as described may be best described as polychotomous, and in some cases polychotomous variables can be ordinal variable. Now I won't get into this debate, rather I suspect that multi-collinearity issue might be there, hence, if exploratory factor analysis has trouble running, try it again with principle component factor extraction with "promax" rotation (if you use SPSS, it is in the factor analysis menu, under "rotation" button). It might help you get a pattern matrix and then go from there. 

There is much written on this under the label "item response theory". The choice of which to exclude will depend on the purpose of the test. For example, if you are trying to measure whether children have reached some particular level, then your items should provide more information at that point in the scale. If you want to measure accurately throughout the span of the ability. Once we know this (please tell us!), then this list can address what to do with the don't know response. There are IRT methods for responses with more than two-categories.

As far as eliminating the two questions everyone answered, some testing companies begin with easy items to try to relieve anxiety a bit, but if almost everyone answers the these items correctly and you are not interested in discriminating at the low end of the scale, it is probably not worth including these in your ten. How big was the sample you used for this, and was this sample similar to the group that will use the test?