Now I have the comments of several products in amazon, and I want to apply the lda model. Firstly, I have to construct the word-document matrix. But it is hard for me to make it, becauce I find it hard to select the word. Although I'm school of statistics and Ihave learned java by myself, I don't know how to remove different signs such as","and "%%", and also remove the stopwords. Can anyone recommand me some books or some materials? Thank you!
Hi,
First of all, some (and sometimes most) of those signs are meaningful. It depends on your application and usages. Other than that, I would suggest you to learn how to tokenize the text into sentences or tokens, and then using different list of stop words that are available on the internet, you can remove them. About the signs, if you only want to clean the text and only keep the tokens (terms ) that contains the alphabets (and maybe numbers), it is easy and you can use regex in Java to replace them with space or nothing. In this way, the terms will remain in the text and then you can use the regex to split them and use them to create you term-document matrix.
Regards
Hi Chen,
If you explain it with more details, I will give you the simple java code to clean your text artifacts.
Mostafa
You can do this easily by using regular expression. You should learn about it.
Better you use this very useful tool (Rapidminer) just googling and download. i have done using this for text mining and it found very useful in text pre-processing such as term stemming,superflous symbol, punctuation symbol and stopwords removing. you just need to learn a bit about it. it has 2 versions opensource (java based) and application. refer many tutorial over youtube about rapidminer..hope it help
I believe that would be easier for you to use a data analysis/statistic tool for this job. Examples are RapidMiner, Knime, Weka, R system (you'll need additional plugin for R), Statistica etc. Have a look at this survey [1] to learn more about methods and features supported by different tools.
[1] http://www.ieat.ro/wp-content/uploads/2013/05/technical_reports/IEAT-TR-2011-1.pdf
Hi,
i am also agreed with Daniel Pop 's statement. if you are interested then i would help you to fix up your problem in R. you have to reply me then on weekend i will give you tips to fix everything in few line of code using R. If you want to do it own-self then please review TextMining using R.
Take care
Bye.
Regards
Muddsair
@Chen
If you use Java and want to remove different symbols such as %, &, @ etc.
simply use this command
str = str.replaceAll("[%&@]","");
what are in square brackets is a regular expression, any symbol of them will be replaced by empty string
you can add more symbols but take care if they already used in regular expression you need to add \ before it
for example if you want to add $ use this command
str = str.replaceAll("[\$%&@]","");
@Chen
for removing numbers
str = str.replaceAll("\d","");
\d means a digit
str = str.replaceAll("\d+","");
for more than a digit
hope it helps
@Chen
for more information, how to use regular expression with Java, kindly take a look at this report
https://www.researchgate.net/publication/272583150_Regular_Expressions_in_Java
Technical Report Regular Expressions in Java
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining