Step# 1: I have collected textual stories from 102 participants and then computed text metrics like # of words, unique words, long words etc.
Step # 2: Then I asked another group of participants to evaluate the creativity on two 7-point likert scale (originality and practicality).
Step # 3: First I computed the inter class correlation (ICC) among all participants and then I drop all stories where ICC was low.
Step # 4: I computed the mean of originality and practicality from scales and finally took the mean of originality. and practicality. to find outcome variable "creativity".
Now my input variables are: word count, unique words, long words and few others.
outcome variable is creativity (scale)
N = 256
I used multiple linear regression with step-wise method in SPSS and obtained the results listed in the image below
whether my results are OK to report in a paper because I have used anonymous participant to gather data through crowd-sourcing platform.
Please I need answers with references from literature that I include in my paper.
Anonymous participants is not a problem. And an R-Squared of 0.25, which means that 25% of the variance in creativity scores has been accounted for, is quite respectable - except that there may be a couple of issues with your methodology. I'll suggest what I see and let other, wiser heads say whether the problems are major, minor, or all in my head.
First: I presume you mean that you calculated the intraclass (not interclass) correlation coefficient. Fine. But the ICC is used mainly as an index of inter-rater reliability. So it would have been the participants judging the stories, not the ones writing the stories, who were evaluated.
Under those conditions, it would be okay to discard judges who failed to agree with the rest, up to a point. The issue here (not a fatal one) is that by tossing out a number of judges, you may be capitalizing on chance agreement among those that remain. How big a problem is this? It depends on what proportion of the judges were tossed out (and you MUST report this fact in your paper!). If, say, you started with 10 judges and just 2 of them failed to agree adequately with the rest, you'd be justified in removing them from the data set. But if, say, 6 of the 10 were discarded, then a careful reader would wonder why there was so little agreement and whether the 4 that remain just "happened" to cluster together.
Second: Much depends on how many predictors you have in your regression equation. If you achieve R-Squared of .25 with just (say) 2-3 predictors, you're doing very well! But if you've got 12 predictors in the equation, it isn't so impressive.
Third: You say you used a stepwise regression procedure to derive your model. Many researchers are very "down" on stepwise regression because it inherently capitalizes on chance. You're adding variables based solely on whether they happen to enhance predictive power in the sample you're studying, which invariably results in "overfitting." This means that the equation you derive will not generalize well.
There are two questions here:
Is 25% explained variance sufficient to report for publication?
Yes!
25% explained variance is quite normal in social research. No, you won't be able to forecast with any reliability, but that's not the purpose of the research is it? You're trying to see if there is a relationship between respondents' word-count, unique words, etc. and perceptions of those respondents' creativity. You have found that if someone writes more, and uses more "triggers" (whatever they are), then readers are more likely to see him/her as creative.
Think about the nature of your data. There is likely to be a lot of noise in your measurements: Creativity is measured with 1-7 rating scales, which are pretty clumsy (even though there are few, or no, well-understood alternatives); word count, etc., are accurate but there is judgement made on what criteria to measure; Many other factors are at play when people respond to your questions on the crowd-sourcing platform - mood, time available, whether respondents were paid, personal importance of the subject matter to each respondent, and much more are likely to affect the amount of effort, and creativity, that people put into their responses. With all those additional influences not included in your model, you should conclude that 25% explained variance is actually rather good.
Are your data valid when your participants are unknown to you?
Yes!
Why would they not be valid? Almost all survey research is, or is supposed to be, anonymous. Anonymity makes your data more valid, not less. Think about some possibly sensitive social or political issue... how might you answer questions if you know that someone might come back to you and ask you to justify your responses? How might you answer if you are confident that the researcher can never know who you are?
I sit on my university Research Ethics committee. I can assure you that we reject any application for a survey when there is any risk that the identities of respondents can be found.
Another issue...
Be careful about using Stepwise Multiple Regression. When you use Stepwise, then you are asking the computer to find the combination of possible predictor variables that gives you the highest R2. You're not testing any theory, and you're letting a machine make decisions for you. If a candidate independent variable is highly correlated with one, or a combination of, other variables that are already in the model, then it will not be selected for inclusion, even if logically it should be.
Stepwise is okay if you're looking for a prediction rule, and you have no interest in why. But if you're looking to explain Creativity, then you are likely to have some ideas (theories) based on your literature review, etc., about what independent variables are most likely to contribute to creativity.
Of course this can expose some other problems: multicollinearity is likely to give you a model where some IV's appear to be not significant, or even with the wrong sign. Don't worry about that (look at the total amount of explained variance, or change in R2).
There is a software product called Grammarly. I have the free of charge email version which gives me a report every week telling me that I have used x% more unique words than other users, etc. May be worth investigating. Hope that helps. I have a sneaking suspicion other software may be out there for textual analysis.
Stepwise is a very real problem because it is unstable as well as because you want an explanatory model. see the attached papers. The R^2 is not a problem. Best regards, David
I wish I could double/tripple thumb up for all answers above but ResearchGate won't allow me for sure. I hope someone else might help me in fulfilling my desire :)
Thanks all for amazing and wise ideas!
I would appreciate if someone could answer the alternate solution of Multiple Linear regression with stepwise method from the following link. Which approach is more suitable
https://stats.stackexchange.com/questions/13686/what-are-modern-easily-used-alternatives-to-stepwise-regression/13698#13698
- Badges
- Science topic
- Similar topics
- Chemistry
- Biochemistry