the way of merge a new machine learning algorithm in spark for big data analytics
adding a new algorithm under pyspark.ml.classification and use it in analyzing big data
As for Machine Learning Algorithms for better analytics, you can use Spark with SVM (Support Vector Machine) and RFT (Random Forest Tree)
As for references: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
#########################################################
No Doubt Spark is a Good choice for big data analytics. But instead of directly jumping on to Spark, I would suggest you to start with the basic building blocks first, so that Spark can be properly understood and implemented.
In my view, starting point for big data is
MapReduce/Hadoop (Welcome to Apache™ Hadoop®!): Apache Hadoop/MapReduce will be a nice entry point and a very basic building block for all high level analytical and processing tools. It all start with Google MapReduce and then an Open Source Implementation with Apache Hadoop. MapReduce is a paradigm for writing distributed code. Hadoop/MapReduce nicely fits as a starting point for the whole Big Data stack.
HDFS (Hadoop Distributed File System) (HDFS Architecture Guide): HDFS is again the Open Source implementation of Google File System (GFS). HDFS and GFS both are used to store files/data across large no of machines in a distributed fashion. If you are thingking where HDFS fits? Well Hadoop mostly consumes data from HDFS and Google MapReduce consumes data from GFS.
Spark (Apache Spark™ - Lightning-Fast Cluster Computing): After understanding above 2 building blocks, it will be much easier to understand and implement Spark. It is one of the emerging platforms and Apache Top Level Project right now. It gives more power than MapReduce/Hadoop. Spark is basically develop to overcome MapReduce's shortcoming that it is not optimized for of iterative algorithms and interactive data analysis which performs cyclic operations on same set of data. For this Spark depends on Resilient Distributed Datasets (RDDs) as a base unit
After clearing the basics of Spark, it also offers 4 high level projects on top of Spark, that are MLlib (MLlib | Apache Spark) a machine learning library, GraphX (GraphX | Apache Spark) a distributed graph library, SparkSQL (Spark SQL | Apache Spark) which is an SQL like implementation on top of Spark and Spark Streaming (Spark Streaming | Apache Spark) to handle continuous flow of data in the form of stream. These sub projects of Spark then can also be used for analysis based on the requirement.
So, to conclude my answer, again I would say that Yes Spark is a Good Technology for Big Data, but its better to start from the Building blocks Hadoop and HDFS and then get hold of Spark.
######### ###############################################
- Badges
- Science topic