The criterion variable was also a composite score.

All perceptions and related words are composites of more basic variables, starting with the physical photos and molecules that activate our sensors.  In other words, all that we perceive, discuss, and write about are abstractions -- the only difference being that some are more abstract than others.  (See Alfred Korzybski's writings, such as "Science and Sanity" for details).  That being the case, since all variables are composites of one sort or another, and given the limitations of such and our present approaches, the use of your composite scores to estimate effect size seems OK for your data.

I can't imagine why using a composite score would influence your ability to measure the effect size. You simply need to calculate an r class or a d class effect size with the data you have.

It seems like looking at the correlation between your composite leadership score and and the appropriate composite PLE score would give you a very clear effect size, for example.

Thank you David.  That was what I was thinking.  However, after talking to my committee member I understand what he was thinking as well.  We would receive more data on the meaning of the construct if we, once again, broke down the construct into various components and created another composite score and used it as a predictor variable.  I did notice (as I unofficially did what you said above) that almost all of my constructs turned predictor variables showed significant success, which was what the committee member was trying to explain to me.  

This is my first dissertation.  I am sure he is trying to keep this project within realm of my ability to graduate soon and also to maintain the quality of the work.  And thank you for stating that it was possible as I may very well calculate r or d and use them for effect size on my own in the future.  

Richard - your response brightened my day.  Quite true actually. I appreciate where you are going with this.  I chose not to use standardized achievement scores because I did not want to promote standardized achievement tests.  So, you are spot on the way that I was thinking!  

Hi Christina,

So you want to measure one latent variable's effect on another latent variable.  Don't worry, you can do this. There's no reason you can't set a criterion level for a latent variable. As Richard says, almost all variables in social science are latent. You just want to make sure it meets some good metric standards.

You already have a composite score, and you want to know if it is suitable measure to make correlations with. You've already concluded your reliability is good. Great! Now you want to test the measurement invariance of your composite scores to find out if your factors provide the same contribution across different groups or situations. You'll have to define those groups or situations in a way that's relevant to your research question.

There are different levels of invariance, with different implications. The level you need depends on what you are doing. If you scale fails your measurement invariance test, you may have to discard one or two items from your scale, but that probably doesn't mean the whole scale is worthless.

Christina,

This paper by a statistician at Sloan-Kettering suggests a rationale and procedure for determining the effect size of a composite score:

Hope it helps.

Don

Thank you so much Don.  I will spend some time going over it.  

You need to use multi variation method. In this way you can use the MADM and ELECTRE methods involving the Shannon's Entropy.

For more information please leave me a message expressing the title of your study exactly.