I'm working on a data set for order classification. This project will be my first confrontation to parallel data mining. I need some help on this. I have found some software like WEKA, R Studios and RapidMiner but I think the software needs to have a parallel nature in order to get most out of multi-core processors. I'm open to any suggestion on getting most out of this computational power specially using WEKA or (mlflex,spark,other,software or new method)
It is not required to have a software with parallel nature. The execution time will be only longer without affecting the data mining results. If you are working with big data (Thousands of gegabytes and more), having a parallel solution on Haddop can be a good solution. In addition, SQL Server has this parallel nature to leverage the power of all parallel processors to have a fast execution time for medium and large databases.
Using Matlab you can Speed up your code based on pre-allocation, vectorization, parfor, spmd....etc.
PRtools is a Matlab based library which is very easy to be used.
personally, I use Weka, it is written in Java and therefore your implementation can be written in different parallel and distributive ways.
Hi Emin,
In my opinion, I recommend you to use R for Data mining, you will find a lot of packages that contain methods covering classification, clustering, association rules, ... etc, using different models. in addition to some helpful functions for data preparation.
easy to use, general, for any thing you want just include the needed package, very good for data visualization.
as Mr.Adrian mentioned Caret package is a helpful one, it includes many classification models, take a look at this Cheat sheet:
https://github.com/ziadbkh/Data-Mining/blob/master/Cheat%20Sheets/Caret%20Library.pdf
Good luck
Firstly, thank you, everyone, for your support, your quick responses and excellent assistance.I tried to use various classifiers in Weka. (I am using sparse matrix methods.) While getting familiar with Rotation Forest in Weka, I stumbled on an Overflow error. (I am using one pc and I changed the heap size in Weka as 6 GB)
I want to solve problems using Weka-Parallel. Weka-Parallel is more complicated because of its two worlds: port number and products required updated. I used R with KNN algorithm but I didn't used to do that with Rotation Forest. As you mentioned, that I can use R with Rotation Forest.
Best Regards
Emin Merhaba :)
bende R tavsiye ederim, öğrenmesi ilk etapta zor ama paketleri çok zengin, her istediğini bulabiliyorsun, selamlar.
Along with Weka tool you can use the Hadoop tool for big data analysis
Different software has different advantage. I like to use Python. But I also think Matlab, C++, R ... are all good tool in big data area.
Dear Emin,
You could use Rhadoop.
Try to follow the contain of this resource if you read french.
http://www.math.univ-toulouse.fr/~besse/Wikistat/pdf/st-tutor5-R-mapreduce.pdf
Good luck.
Dear Emin,
You could use R for your data mining studies. you will find alot of packages regarding data mining.
R and Python are leaders of the pack , as far as opensource is concerned. While R has a great academic following, Python is more accepatable commercially. Both have wildly growing communities. It may be better to keep the data storage and modeling part separate, R, Python both offers some integration with Hadoop!
Thank you so much, all of you, for helping me move forward.
Best Regards
Dear Emin,
Please take a look at this study: https://www.datacamp.com/community/tutorials/r-or-python-for-data-analysis.
I recommend R with Hadoop for Big Data Analysis.
There are some tools for stream mining:
WEKA
Orange with python
Ggobi
Tangara
Knime
MOA, etc.
I hope that this information could help you.
Best regards.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining