如何进行Spark中MLlib的本质分析

org.apache.spark.ml.attribute
org.apache.spark.ml.classification
org.apache.spark.ml.clustering
org.apache.spark.ml.evaluation
org.apache.spark.ml.feature
org.apache.spark.ml.param
org.apache.spark.ml.recommendation
org.apache.spark.ml.regression
org.apache.spark.ml.source.libsvm
org.apache.spark.ml.tree
org.apache.spark.ml.tuning
org.apache.spark.ml.util

org.apache.spark.mllib.classification
org.apache.spark.mllib.clustering
org.apache.spark.mllib.evaluation
org.apache.spark.mllib.feature
org.apache.spark.mllib.fpm
org.apache.spark.mllib.linalg
org.apache.spark.mllib.linalg.distributed
org.apache.spark.mllib.pmml
org.apache.spark.mllib.random
org.apache.spark.mllib.rdd
org.apache.spark.mllib.recommendation
org.apache.spark.mllib.regression
org.apache.spark.mllib.stat
org.apache.spark.mllib.stat.distributed
org.apache.spark.mllib.stat.test
org.apache.spark.mllib.tree
org.apache.spark.mllib.tree.configuration
org.apache.spark.mllib.tree.impurity
org.apache.spark.mllib.tree.loss
org.apache.spark.mllib.tree.model
org.apache.spark.mllib.util

DataFrame: Spark ML uses DataFrame from Spark SQL as an ML dataset, which can hold a variety of data types. E.g., a DataFrame could have different columns storing text, feature vectors, true labels, and predictions.
Transformer: A Transformer is an algorithm which can transform one DataFrame into another DataFrame. E.g., an ML model is a Transformer which transforms DataFrame with features into a DataFrame with predictions.
Estimator: An Estimator is an algorithm which can be fit on a DataFrame to produce a Transformer. E.g., a learning algorithm is an Estimator which trains on a DataFrame and produces a model.
Pipeline: A Pipeline chains multiple Transformers and Estimators together to specify an ML workflow.
Parameter: All Transformers and Estimators now share a common API for specifying parameters.

Classification
	Logistic regression
	Decision tree classifier
	Random forest classifier
	Gradient-boosted tree classifier
	Multilayer perceptron classifier
	One-vs-Rest classifier (a.k.a. One-vs-All)
Regression
	Linear regression
	Decision tree regression
	Random forest regression
	Gradient-boosted tree regression
	Survival regression
Decision trees
Tree Ensembles
	Random Forests
	Gradient-Boosted Trees (GBTs)

K-means
Latent Dirichlet allocation (LDA)

Local vector
Labeled point
Local matrix
Distributed matrix
	RowMatrix
	IndexedRowMatrix
	CoordinateMatrix
	BlockMatrix

Binary Classification: linear SVMs, logistic regression, decision trees, random forests, gradient-boosted trees, naive Bayes
Multiclass Classification:logistic regression, decision trees, random forests, naive Bayes
Regression:linear least squares, Lasso, ridge regression, decision trees, random forests, gradient-boosted trees, isotonic regression

K-means
Gaussian mixture
Power iteration clustering (PIC,多用于图像识别)
Latent Dirichlet allocation (LDA，多用于主题分类)
Bisecting k-means
Streaming k-means

DecisionTreeModel
DistributedLDAModel
GaussianMixtureModel
GradientBoostedTreesModel
IsotonicRegressionModel
KMeansModel
LassoModel
LDAModel
LinearRegressionModel
LocalLDAModel
LogisticRegressionModel
MatrixFactorizationModel
NaiveBayesModel
PowerIterationClusteringModel
RandomForestModel
RidgeRegressionModel
StreamingKMeansModel
SVMModel
Word2VecModel

import org.apache.spark.ml.classification.LogisticRegression 
import org.apache.spark.ml.param.ParamMap 
import org.apache.spark.mllib.linalg.{Vector, Vectors} 
import org.apache.spark.sql.Row 

val training = sqlContext.createDataFrame(Seq(   (1.0, Vectors.dense(0.0, 1.1, 0.1)),   (0.0, Vectors.dense(2.0, 1.0, -1.0)),   (0.0, Vectors.dense(2.0, 1.3, 1.0)),   (1.0, Vectors.dense(0.0, 1.2, -0.5)) ))
    .toDF("label", "features") 
val lr = new LogisticRegression()
println("LogisticRegression parameters:\n" + lr.explainParams() + "\n") 

lr.setMaxIter(10).setRegParam(0.01) 
val model1 = lr.fit(training) 
println("Model 1 was fit using parameters: " + model1.parent.extractParamMap) 

val paramMap = ParamMap(lr.maxIter -> 20)
    .put(lr.maxIter, 30)
    .put(lr.regParam -> 0.1, lr.threshold -> 0.55)
val paramMap2 = ParamMap(lr.probabilityCol -> "myProbability") 
val paramMapCombined = paramMap ++ paramMap2
val model2 = lr.fit(training, paramMapCombined)
println("Model 2 was fit using parameters: " + model2.parent.extractParamMap)

test = sqlContext.createDataFrame(Seq(   (1.0, Vectors.dense(-1.0, 1.5, 1.3)),   (0.0, Vectors.dense(3.0, 2.0, -0.1)),   (1.0, Vectors.dense(0.0, 2.2, -1.5)) ))
    .toDF("label", "features")
model2.transform(test)
    .select("features", "label", "myProbability", "prediction")
    .collect()
    .foreach { case Row(features: Vector, label: Double, prob: Vector, prediction: Double) => println(s"($features, $label) -> prob=$prob, prediction=$prediction")   }