Sara I hope the following does match your inquiry about reading text difficulty:

https://texteval-pilot.ets.org/TextEvaluator/

This tool is by ETS and requires login info.

The following is free and requires no login info, it also provides some info on to whom the text more suitable 

http://www.renaissance.com/Products/Accelerated-Reader/ATOS/ATOS-Analyzer-for-Text

I hope this helps.  

Hi Sara,

I suggest you locate the question by Jan Philipp Thomecsek: "How can the complexity of language be measured?". You'll find a number of good answers there. Let me know if the thread is closed, because, as a contributor, I have all the answers on my page. It should still be going though.

Dear Sara,

I think  you should try Lexiles, a scientific approach for matching students with appropriate texts. It provides a common scale for measuring text difficulty and student reading ability.

Good luck with your endeavor.

Best regards,

R. Biria

Hi Sara,

you could perhaps also use the Flesch-Kincaid readability score, which is supposed to give a measure indicating the difficulty of a reading passage in English: https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests

There are various tools online that compute it, such as https://readability-score.com/

Hope this helps!

Cheers,

Paolo

Thanks everyone, your answers are very useful. However, it looks like tools you're suggesting are english-oriented, but I'm going to work with Italian and Slovenian. In particular, I have to make sure that texts in these languages have the same complexity. Do you think that Gunning Fog index is reliable? I've read many critics in the literature.

Both Gunning Fog index and Flesch Kincaid index can be applied to any language. I can't say much about their reliability, anyway.

For Italian, you may also want to have a look at this: https://it.wikipedia.org/wiki/Indice_Gulpease

Paolo

Hi, Sarah,

I use Item Response Theory to measure the items I create.  It helps to match students' ability with item difficulty.  I've learnt a lot from the answers to your questions, though.  So thks everyone.

Orieta

If you have word frequency lists from children's book and grade school text books, you can also evaluate how vocabulary use for each article compares against the frequency lists that represent more general language use.

You can also measure lexical dispersion.  Here is a post explaining the concept and usage though the data is from Chinese.  You can follow the references to get more information.

http://www.zhtoolkit.com/posts/2013/02/improving-character-frequency-lists-with-dispersion-data/

Check out the web-based lexical complexity analyzer at http://aihaiyang.com/software/

Hi Sara,

I'd also recommend this article:

Lu, Xiaofei (2010), Automatic Analysis of Syntactic Complexity. International Journal of Corpus Linguistics 15(4), 474–496.

It also evaluates English, but discusses 14 different text based metrics and addresses syntactic complexity in particular, which is ignored by vocabulary based measures. For some of the methods you'll need to parse the data, but at least for Italian that should be possible (not sure about the situation for Slovenian, I think there's a dependency parser for it though).

Hope this helps!

Amir

I once attended a lecture by Benedikt, who intriguingly uses compression programs (e.g., Zip) to approximate Kolgomorov complexity; in this way he is also able to compare between languages. More information: http://www.uclouvain.be/cps/ucl/doc/cecl/documents/Benedikt.pdf 

Hi Sara,

I know exactly how you feel difficulty. Majority of L2 performance research takes English as a target language so that it's a little bit tough to measure linguistic complexity in other languages. So, I believe that we should go back to our theoretical consideration. In other words, how do we define the construct, complexity? And then, we can pick up appropriate measures for that construct.

As for linguistic complexity, several measures are available such as discourse complexity, syntactical complexity, morphological complexity (different kinds of tense and modals), and lexical complexity (diversity).

Good luck!

Shungo

Thanks to Haiyang and Amir's posts. I just used Xiaofei Lu's article (and others on his website) in a class Language Teaching Technologies, and tested Haiyang's web tools to evaluate syntactic complexity. It's great. Students are very interested.

Sara,

Thank you for the post and everyone for helpful answers as well as resources. I've learned a lot, and tested a few algorithms and sites.  

In the spirit of measuring text complexity in any language, using whatever resources that's available, and/or build good-enough resources quickly without spending a life time on it and forget about your original goal, we can minimally start with the Flesch-Kincaid readability tests, which only use average syllable count per word and average word count per sentence. If you are working with a written language that indicates word boundary (like using a space), you are good to go.  If you are dealing with a written language that doesn't indicate word boundaries, like Chinese and Japanese, then you'll have to do word segmentation first.  There are a lot of people working on it (I worked on it before there were any on-line resources for Chinese) and there are many open source programs to do it. So consider the word segmentation problem solved (or good enough to use).  Plug in the number, and you'll have the Flesch-Kincaid readability score:

0.39 *  average_word_count_per_sentence + 11.8 * average_syllable_per_word - 15.59

To count the number of sentences you have in the text, think about the punctuation marks that separate sentences, such as ". ? !" and break your text by these symbols. It's not always right, for example, in English "." is ambiguous. It is also used to mark abbreviation, among others. So this is an estimation. 

If you don't want to write a syllabification program, you can approximate it by replacing all consonants in the writing with a white space and count the number of vowels. This is a good-enough approximation to count syllables that takes just seconds and doesn't require any programming. 

The Flesch-Kincaid formula is the results of a multiple linear regression model with two terms and an intercept (-15.59).  If you have a corpus of texts already classified into readability levels, you can easily train your own algorithm, adding terms that you can estimate. There are so much more resources available now than Flesch-Kincaid's time, so even for languages without too much resources, this goal is easily within reach.

The next level, I think, is to add frequency information.  Otherwise, "I am Sam, Sam I am." will have the same score as "Life is hard. We are doomed."   Sara needs resources for Italian and Slovenia. There are many Italian frequency lists and corpus and there are open source ones.  For Slovenia, here is a simple lecture on how to use the web to create resources for minority languages, and Slovenia is used as an example.

cikm01-talk.ppt - Carnegie Mellon University

I would also like to add that Linguistlist.org is working hard to address this question, adding best practices, resources, tools and general information as we talk.

Next, there are works illustrated by Xiaofei Lu and Haiyang Ai (see Haiyang's earlier post) that uses a parser to get part-of-speech tagging and syntactic structure to add more features to estimate text complexity.  To do this you'll need a parser, to train a parser you'll need resources like the Penn Treebank. A quick search of the Linguistic Data Consortium (LDC) on tree turns up a whole page of Treebank databases. You have text corpora with part-of-speech tagging and syntactic tree annotation in, say, Arabic, Chinese, Czech, Dutch.  Find a parser (Stanford is a good place to start) or a colleague who knows how to do it, and you can add features (Lu and Ai's papers have a lot of good ideas).

SALT (Systematic Analysis of Language Transcripts) can be used with bilingual Spanish/English speakers and on their website, it looked like perhaps French and Turkish.  I don't have any personal experience with using SALT with people who speak a language other than English.

Hello Sara,

would you be interested in grammatical complexity other than lexical complexity, you could have a look at Mean Length of Utterance (MLU). 

You may want to have a look at the CHILDES database (vid.nb1). There you can find samples and methods for spontaneous speech of child language. The site also has information on other centres and initiatives on your topic, and there is even a section on bilingual acquisition (vid.nb2)! You could also have a look at the CLAN programme to analyse your data (vid.nb3).

Regards,

Adrià

nb1. http://childes.psy.cmu.edu/

nb2. http://childes.psy.cmu.edu/manuals/04biling.pdf

nb3. http://childes.psy.cmu.edu/clan/

Hi, Sara

Have you seen this:

Laura B. Casey, Neal D. Miller, Michelle B. Stockton & William V. Justice

(2016) Assessing Writing in Elementary Schools: Moving Away From a Focus on Mechanics,

Language Assessment Quarterly, 13:1, 42-54

http://www.tandfonline.com/doi/full/10.1080/15434303.2015.1136311

Thanks,

Hugh

Type/Token ration is a less partial measure in comparing texts in different languages. It is a better measure, espcially while the texts to be campared are from typologically disticnt families (such as Sinitic languages and Roman languages).