Hello everyone,
I would like to perform machine learning and deep learning in a near futur on clinical data, coming from different sources and technologies (Omics data, Imaging data, and so on).
These data will have to be stored in a structured way in a datawarehouse and will be accessed thanks to a 10Gbit network and I will have some clusters available.
My question is the following : What could be the best architecture to use to run parallel calculations/deep learning/machine learning on these data ? I first though at Spark but would it really be appropriate to wide data (often few observations for lot's of variables - thousands to millions biomarkers analyzed) ?
Is there other framework(s) more adapted to this kind of question ?
What kind of advices would you give me to stick to the best "big data" practice ?
If Spark seems the good answer to my problem, would you use R language (as library sparklyr) or Scala/Python instead to perform deep learning/machine learning ?
Thanks in advance for all your advices,
Best regards
Big data are complex to use and deep learning provides solution for it by extracting high-level, complex abstractions as data representations through a hierarchical learning process.
SparkR (R on Spark) and Scala/Python both are used depending on what are you looking for.
I recommend look into
1. JAVA - WEKA - http://www.cs.waikato.ac.nz/ ml/weka/
2. Python - Scikit Learn - http://scikit-learn.org/ stable/
3. R caret Package - https://topepo.github.io/ caret/index.html
I have attached link here for you with list of other tools for big data analyses
https://www.import.io/post/all-the-best-big-data-tools-and-how-to-use-them/
Big data are complex to use and deep learning provides solution for it by extracting high-level, complex abstractions as data representations through a hierarchical learning process.
SparkR (R on Spark) and Scala/Python both are used depending on what are you looking for.
I recommend look into
1. JAVA - WEKA - http://www.cs.waikato.ac.nz/ ml/weka/
2. Python - Scikit Learn - http://scikit-learn.org/ stable/
3. R caret Package - https://topepo.github.io/ caret/index.html
I have attached link here for you with list of other tools for big data analyses
https://www.import.io/post/all-the-best-big-data-tools-and-how-to-use-them/
If you just want to analyses data; weka will be good option. but if you want to use these data in real you will be suppose to apply hadoop
I would use TensorFlow and convolutional neural networks. They have been very effective on medical imaging such as organ detection, tumor classification etc. You can use TensorFlow with Spark.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python