hi guys, i am working on dataset with 1M user and 943,347 item , but i want to consider less than, for example 10,000. my platform is weka. my reason is that ram of laptop can not process on 1M. i need a strong reason that why i consider less of data?
My work is not on big data. i want to know this number (1M) is big data, or need parallel systems or distributed systems? thanks
Upfront disclaimer - I don't work in "big data" either. But when I'm asked about what is big data, my smart aleck response is "it is big data when I can no longer do it on my laptop with weka." But really, its a term that lacks a formal definition and is rather a concept. To me, it is the case where you have more information than you are able to deal with using the tools you are comfortable with. It is a relative term.
Parallelism may or may not be an option, and even if it is an option, it may not be a solution. Weka is a large suite of algorithms so depending on which parts you are using, you may not be able to scale out (again, disclaimer, not a weka expert so I'll not be much use advising what methods in weka *could* run in parallel).
Are you saying you have one million entities, each with ~950,000 attributes? In that case you are suffering from the "curse of high-dimensionality". Dimensionality reduction is likely in order before you are going to get anywhere.
Hopefully this helps give you some perspective, sorry though I'm not able to present a clear road forward.
-mike
hi, thanks . maybe i asked my question bad, my meaning is, i want to consider less than user because of single computer can not process 1M data. and i need distrusted systems. therefore, i need a paper for referencing in which mentioned single machine can't process big data and for processing it , we need distributed systems. do you know any paper that i can referencing it ?
Hi
simply I have a work withdrawing 40 millions expressions from a PDF file to make analyze and get the reason of statistics basing on Tweeter , the first time I was thinking the same that I have to divide the file on pieces and share those pieces between some computers which they are going to launch the process in parallel but I have found a that there is a possibility to make that in one computer just I used a program to withdraw all the expressions and after every creation of the process and getting the expression the program automatically refresh the memory to be empty.
then as I understood try to perform your task using features from a oriented object language such as : JAVA or C++
best regards
Abdou
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data