I am currently working on some real-life market data. The dataset consists of around 1200 samples with 20 attributes describing each sample. I am facing two problems:
1) The dataset is a text dataset. I am facing a lot of problems in parsing the data. Could anyone recommend some methdology to handle the data.
2) Standardization of dataset: The attributes are not standardized e.g. In Colur, there are values such as "Blue, Black" for one sample and "Black, Blue" for another. How do I go ahead with this?
If you want to parse natural language text data and convert it into bag-of-words representations for some machine learning purpose, I have used TagHelperTools for a similar type experiment
http://www.cs.cmu.edu/~cprose/TagHelper.html
Basically it wraps text analytics functionality provided by NLP processing packages such as the Stanford CoreNLP toolkit in a relatively simple to use format taking xls or csv input and outputting Weka formatted ARFF files which include part-of-speech tags, word frequencies and other statistics.
Do you wish to use textual features from the data to recommend certain samples?
A couple questions: Does the data have *any* kind of structure to it: json, xml, something else, or is it really just raw prose? Also, what programming languages are you familiar with? There are several tools and libraries that would be useful in processing this data (I can think of a half a dozen in Python), but I wouldn't want to suggest something overly complicated...
The most important question is: What are you looking for? Do you just want to count (co-)occurrences of specific words (syntactical), or do you need to go further and interpret (more semantical)?
There is a wide spectrum of "pre-processing" methods. What is appropriate depends to a large extent on what you want to achieve during processing. However, there are also some rather simple methods, such as stemming [1] and removal of stop-words (language-dependent), which may always apply.
On the more pragmatic side, if you just need a way to store a text-database with a flexible schema (what you call "non-standardized attributes"), I'd recommend using a non-SQL database such as mongoDB [2]. This would allow you to write easy queries once the data is imported. There are also tools to import text data to mongoDB [3].
[1] - http://en.wikipedia.org/wiki/Stemming
[2] - http://www.mongodb.org/
[3] - http://docs.mongodb.org/manual/reference/program/mongoimport/
Python! Has great packages for parsing text and pre-processing data for machine learning algorithms.
Gerrard and Dennis, thank you for your replies. I basically wanted to convert my data to some numerical form, which I am used to handling. OR Finding out the base words to construct a dictionary of sorts in order to process my data. I will proceed on the lines you have mentioned.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining