Does anyone know about some existing stop-word vocabularies ?
I am interested in doing some keyword text mining work and I was wondering if there are some existing stop-word vocabularies that I can use to reduce the noise from the data.
If you need more lists, Google for:
a about above after again against all am an and any are
https://www.google.com/search?q=a+about+above+after+again+against+all+am+an+and+any+are&oq=a+about+above+after+again+against+all+am+an+and+any+are
Hi,
In addition to the first answer:
you can find stopword lists for several languages in the page of snowball stemmer: http://snowball.tartarus.org
for example for french: http://snowball.tartarus.org/algorithms/french/stop.txt
Also, you can generate stopword lists based on frequency lists extracted from corpora. Wiktionary offers a numbers of lists in various languages extracted from different sources:
https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists
Or you can compile your own list from corpora, using e.g. wikipedia dumps.
Cheers!
There's the stop list used as part of the Rainbow classification system: http://www.cs.cmu.edu/~mccallum/bow/rainbow/
There are various good answers already given. I would like to add that often times stop words are domain specific, so it may be good idea to build a stop word list for your domain and there are few basic approaches available for the same.
This is stopwords list that I made for my projects. If you're using c#, add the class and use as follows:
file_contant = StopwordTool.RemoveStopwords(file_contant);
Using Python NLTK, the whole “stopwords” dictionary contains common stop-words from 11 languages. NLTK is a leading open source platform for building Python
programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet.
See Bird S., Loper E. and Klein E., Natural Language Processing with Python. O’Reilly Media Inc. (2009)
***
if you think this answer is useful, please appreciate it using the green Arrow, thanks
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining