I want to delete stop words in a file. The file contains a Persian sentence, a tab and then an English word in each line. The stop words are in a separate file. The code (python 3) I wrote works but the problem is, it deletes a stop word in some lines but it doesn't delete the same stop word in some other lines. It happened almost for every stop words. I guessed maybe it is because of not normalizing the words. So I normalized the 2 files, but again there was no change in the problem. What can be the cause of this problem and how can I fix it?
The file contains a sentence, a tab and then an English word. to access the sentences I have to split them with tab.
You might want to try using regular expression replace function to do this. You can probably define a relatively simple regular expression that matches these words and you can simply replace every match with an empty string or some other string if you wish. You can load the entire file with the read function without any arguments for this file and than execute a regular expression replace on that string.
why do you clear 'line' here?. it has no sense:
for row in corpuslines:
line = ""
etc...
I don't know python but I think that this way, only the last 'row' in corpuslines will be processed...
- Badges
- Science topic