After stemming and parsing, how do I filter stop words and adjectival clauses from the dataset?
You can have a stop list; write a function to replace those words with space,or whatever you want; apply the function to the whole dataset.
Hi,
you might use your own stopwords file or nltk stopwords for example
stopwords=nltk.corpus.stopwords.words('english')
sentence=[ word for word in nltk.tokenize.wordpunct_tokenize(sentence.lower()) and word not in stopwords]
....
to add the step of tagging you might use python tagger or stanford tagger
see this for example:
text = nltk.word_tokenize("And now for something completely different")
tags_words=nltk.pos_tag(text)
words =[word[0] for word in tags_words if word[1] !='JJ' ]
>>> tags_words
[('And', 'CC'),
('now', 'RB'),
('for', 'IN'),
('something', 'NN'),
('completely', 'RB'),
('different', 'JJ')]
>>> words =[word[0] for word in tags_words if word[1] !='JJ' ]
>>> words
['And', 'now', 'for', 'something', 'completely']
http://www.nltk.org/book/ch05.html
Issa
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages