Hello, Ms. Holloway -

I just recently used that instrument in my PhD dissertation research, which I successfully defended on 11/25, inst. Dr. Bouckenooghe gave me permission to use it freely. His email is: [email protected]

I would think that since institutes of learning are also organizations, even in an academic context, this questionnaire would be appropriate

How big is your sample? I used a sample of 300 and my reliability coefficient was better than the Cronbach alpha that was produced in the development of the instrument. I used only the Readiness for Change section and reliability exceeded .80. Also, you have to remember to reverse score the Cognition items, else your Cronbach alphas will be incorrect.

Since all sections of the instrument displayed alphas that were reliable in original construction of the instrument, you can be assured that it will measure what it is supposed to be measuring. Changes in reliability coefficients will vary depending on sample size, so you have to take that into consideration. If your sample is small, your findings can be biased because of that, and can cause you to accept the null when it should not be accepted (Type I error).

I used a moderated multiple regression analysis (MMR)  test in my study because, according to the literature, since 1977 that  has been one of the preferred tests for organizational studies. However, to have a reliably detectable effect size, the literature advises a minimum sample size of 300 for  MMRs. These are things that have to do more with the design of your study, rather than the instrument itself (although the appropriate instrument is an important aspect of all studies).

Hope this helps..

Ruth

Hello, Paul!

Thank you for some very good observations and questions! I appreciate the opportunity to learn!

Reliability and Validity

When I mentioned reliability, I should have expanded more on my explanation: reliability has to do with the consistency of the measurement instrument and/or design of the study (this has to do with other researchers being able to replicate the study - among other things). Reliability contributes to validity; however, reliability is not dependent on validity.

There are two types of validity: internal and external. Internal validity has to do with how truthful the research is; that is, how appropriate and relevant the design and measurement is to the study, for this will inform the accuracy of the conclusions. It also answers questions such as whether or not the variables that are meant to be tested are really being tested, or if what is being observed or queried is what the researcher meant to be observed or queried (Vogt, 2007). It must be noted that in this way, instrument reliability contributes to research validity;

Your observation on this point was a good reminder to always write from the reader’s point of view, and not assume that the reader will make the mental connections that I take for granted. I really appreciate this!

In my reference to the MMR literature for why I chose the MMR analysis you stated: “Statistical analyses should suit hypotheses, designs, and data, not honor posterity.”

My response: Choosing this type of test (MMR) had nothing to do with honoring posterity, which I would consider to be a choice that would be rooted in subjectivity and not objectivity. My choice had to do with the intent of my study, which was to understand the interactive effects of education and personality traits on the outcome variable. Unlike the processes that are related to other studies that involve moderators (e.g. hierarchical analysis), this particular procedure involves multiplying the moderator by the IV, creating a new interaction term (and other procedures such as mean centering) before running the test. According to the literature, it is a more sophisticated and better suited procedure for studies that probe interactions. However, since I have never heard of anyone choosing a statistical test just to honor posterity (a totally useless endeavor, I would think!) – perhaps that might be an interesting study for someone to explore?

You quoted me in saying “However, to have a reliably detectable effect size, the literature advises a minimum sample size of 300 for MMRs.", and asked two questions, each of which I will address separately, as follows:

You asked: Do you think that this is this true for very strong effects, as well as for very weak effects?

First, I must clarify the matter of sample size in regards to a MMR procedure: that 300 size that I mentioned is in relation to categorical moderators; the recommended sample size for studies with continuous variables is smaller (see Shieh, 2010, p. 828 for a range, and reasons for the range).

Concerning small and larger effects. My study emerged from a Croatian study that used a small sample size of 87 and findings of that study supported the null. The researchers acknowledged the consequence of the small sample size on their ability to draw accurate conclusions and, furthermore, also acknowledged that the findings could not be generalized to the larger population. They recommended a larger sample size; hence my choice to go with the well documented evidence and recommendations from the literature.

Researchers must be aware of all of the various types of biases and effects that can result from sample selection, including sample size bias. Due to the size and heterogeneity of my sample makeup, my results were different from the Croatian study, and, contrary to their findings, the null was not supported because small to moderate effect sizes were detected in my study. These were undetectable in their smaller sample size.

Based on the explanation that I provided concerning the Croatian study and my own study, results clearly showed that a small sample size did not even show a weak effect. Small sample size can bias results so that the null might be supported, when in fact it should not. In MMR analysis that uses categorical moderators, both strong and weak effect sizes can be detected based on a sample size of 300.

You also asked: Or do you think that minimum appropriate sample size varies as a function of anticipated strength of effect, regardless of the statistical procedure one uses?

My response: Hmm… there is some subjectivity implied in your question (“do you think…”), and quantitative analysis has to do with objectivity. My first disclaimer would have to be that I am not an expert in all types of statistical procedures, so I cannot offer an intelligent response to this question. However, as far as multiple regression goes (and there are several types), I stand by my previous comments regarding sample size.

Now, the literature does offer guidelines for choosing a sample size that will afford detection of an effect, presumably across various procedures. Some heuristics are based on 10, 15, 20 – 50 respondents per variable (this is easily confirmed in the literature, so I won’t add references).

Personally, I think that for most researchers, they will consider such things as the purpose of the research, and the resources that are available when considering the design of the study – among other things. They might not be interested in generalizing their findings to a larger population, but only to their sample. In that case, they might be comfortable using the recommended rule-of-thumb (10 – 50 respondents per variable). I have often wondered about how results of studies that are based on heuristics might be used or misused. For example, if a manager conducts a study with say, 50 people, and publishes the results, I wonder how much misinformation can be spread if the results are picked up in non-academic publications such as magazines and news articles, and written in such a way as to imply generalization of the results?

Once again, thank you, Paul, for challenging me and allowing me the opportunity to stretch mentally. I love these types of interactions!

Best Regards,

Dr. Ruth (Yes, I am still wallowing in the joy of the new title… I expect that the newness of it will fade in time – like maybe in 100 years?! LOL!)

References:

Aguinis, H. (1995). Statistical power problems with moderated multiple regression in management research. Journal of Management, 21, 1141–1158. doi:10.1016/0149-2063(95)90026-8

Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90(1), 94–107. doi:10.1037/0021-9010.90.1.94

Shieh, G. (2010). Sample size determination for confidence intervals of interaction effects in moderated multiple regression with continuous predictor and moderator variables. Behavior Research Methods, 42(3), 824–835. doi:10.3758/brm.42.3.824

You can use the instrument by acknowledging the author, and determining the reliability in your context.