Note: So far I have experimented with an untrained TreeTagger, but (unsurprisingly) only with mediocre results :-/ Any hints on existing training data are also appreciated
The results so far can be viewed here: http://dh.wappdesign.net/post/583 (lemmatized version is displayed in the second text column)
Thank you Martin, I will take a look at this. "Facebook" was not meant as a mandatory requirement; what I am really looking for, are good tools and training data to lemmatize computer mediated / social media language (in German).
well, it's not what your are looking for (it wasn't rained on social media data) but maybe it yields better results than TreeTagger: MATE Tool (https://code.google.com/p/mate-tools/downloads/list) with this Model (https://code.google.com/p/mate-tools/downloads/detail?name=ger-tagger%2Blemmatizer%2Bmorphology%2Bgraph-based-3.6%2B.tgz&can=2&q=)
Btw, at a first glance, your treetagger results do not seem that bad. Maybe one can come up with some basic rules to handle smileys or unknown words...
maybe you can check Morphix - http://www.dfki.de/~neumann/morphix/morphix.html
See more details below !
It does lemmatization and inflectional analysis. Also online decomposition of compounds. It is lexicon based, but comes with a huge lexicon!
It is implemented in CommonLisp, and very easy to install and use, see the web page. If you have a Linux (Ubuntu), you can use clisp directly and install and run Morphix.
Usually, the main API work on isolated words, but there is an API function called (split-and-rt-wrds ""), e.g., cutting and pasting the first text of the link you provideds looks like this:
(split-and-rt-wrd "Aber eines muss man sagen: Larissa stand auch heute im Mittelpunkt - allein schon vom Umfang der Sendezeit. Ohne sie wäre es soooo langweilig gewesen! Schade, dass sie nicht gewonnen hat und heute auch nochmal so viele Schläge ins Gesicht bekam... Am liebsten hätte ich ihr ein Taschentuch gereicht und sie in den Arm genommen.... :( LARISSA unsere DSCHUNGELKAISERIN (Sonja und Daniel haben es gerade bestätigt - es ist OFFIZIELL!!!)
Thank you Bastian and Günther, I will try out the tools you suggested.
Hi Manuel,
there are indeed some options you can choose to lemmatize German. In case you are already happy with a stemmer you might want to have a look at this part of NLTK: http://www.nltk.org/api/nltk.stem.html
If you need lemmatization you probably find something useful here if you are familir with python:
https://docs.google.com/document/d/1rdn0hOnJNcOBWEZgipdDfSyjJdnv_sinuAUSDSpiQns/edit?hl=en
If you prefer java you might want to look at Stanfords parser:
http://nlp.stanford.edu/software/lex-parser.shtml
That is to my knowledge also able to parse and lemmatize German.
Python as well as Java have API's that allow you to scrape Facebook just google for it. It is easy to find. I hope this helps you.
Cheers, Markus
Thank you Markus, I am aware of Python NLTK and the Standford Parser. My main problem, however, is that these tools have some issues with German social media language, which oftentimes uses unorthodox spelling or vocabulary. Anyway, I will take a closer look a both suggested toolkits again, thx for the hint.
Dear Manuel,
I guess you have already seen the folowing papers, that were presented at KONVENS 2014 last week and that exactly discuss this problem:
- Melanie Neunerdt, Michael Reyer and Rudolf Mathar. Efficient Training Data Enrichment and Unknown Token Handling for POS Tagging of Non-standardized Texts
- Andrea Horbach, Diana Steffen, Stefan Thater and Manfred Pinkal. Improving the Performance of Standard Part-of-Speech Taggers for Computer-Mediated Communication
The proceedings can be downloaded here: http://www.uni-hildesheim.de/konvens2014/pages/Proceedings.html
Thank you Christian, yes, I am aware of these ;-) Thanks for the hint anyway
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts