I have created custom dictionaries pertaining to my research question, which I am using for text data analysis using LIWC. Can I use Cronbach alpha based on standardized items to check for the reliability of scales?
Linguistic inquiry and word count should be considered as a preliminary and intermediate step in text analysis on the way to the ultimate codification of themes and concepts being represented and communicated by those specific text constructions, finally arriving at an understanding of the semantic network among those codifications as the strength of association within and among those shared network nodes. Cronbach's alpha was originally intended to reflect the agreement among multiple interpretations of semantic network themes and concepts (as reflected in responses from multiple "raters"). On the other hand, Cohen's Kappa is a test of overall and point by point agreement between just two "judges" of the semantic network (text, questionnaire, instrument, tool, survey, poll, etc.) at a time.
If you treat individual text mining algorithms as if they were individual respondents, attempting to interpret a passage of text, you could theoretically use Cronbach's alpha test to index the inter-rater reliability of the material being interpreted rather than the reliability of the algorithms. That could be very interesting to see at what point the increasing complexity of the material being interpreted resulted in a lower Cronbach's alpha index score. On the other hand, if you compared two different algorithms' interpretation of the same passage of text, using the Cohen's Kappa test, you could theoretically directly compare the algorithms to each other. Running both types of tests on the same material is rarely done and could be a very interesting project. Just my opinion.
- Badges
- Science topic