Hello everyone,
I am doing some research on Fake News detection for this I would like to develop a measure that tells the difficulty level of the article. Actually I would like to analyze the webpage content.
Could someone give some good ideas to start with? I am looking for a Machine Learning kind of algorithms to develop the measure?
Note: I have gone through the indexes like ARI, SMOG, etc... but they are just based on number of words/sentences in the article and mostly they do not concentrate on number of complex words.
https://www.ed.ac.uk/information-services/library-museum-gallery/finding-resources/library-databases/databases-overview/evaluating-websites
Please your visit this site. I think this is helpful for you.
Just passing by without much expertise in web analysis, but I think the expanse (total size and broadness -- will be your homework here) and level of vocabulary could be used as well. There are some ways to categorize a word as easy or hard. I can think of comparing the words against the national or reputable database of words, such as government-approved textbooks, or check them against essays written by different level of children in schools. Japanese language has a clear cut categorization of words into levels, because they have national examinations based on this. Other languages may also have a similar case. In this case of comparing against word databases, the words are categorized by level of education and/or age.
Another way is to find the frequency of that word in the general use of the language itself by finding the relative amount of occurrences of that word in a corpus. The problem now becomes choosing a good corpus. These should provide some insights into finding which words are easy or difficult, beyond using the number of words or clauses in a sentence.
Words that are no longer in common use can also be regarded as more difficult, as uncommon or recently "rediscovered" words like "dotard" which only regained spotlight because allegedly that's how Kim Jong Un called (through a translator) Donald Trump. I initially thought it was a made-up word from Donald + Retard, but no, it's a real word that had seen minimal uses (probably minimal outside literature), maybe only until this year.
Again, I am just writing suggestions based on my experiences. Seeing that it has been two days and there is not much attention, I only wish that my answer helps.
Thank you very much Mr. Chawanat Nakasan. Your suggestions gave me some new ideas. I shall for sure consider your suggestions.
- Badges
- Science topic
- Similar topics
- Psychology
- Mental Processes
- Learning