Migrating from Spark 1.6 to Spark 2.0

Other topics

Update build.sbt file

Update build.sbt with :

scalaVersion := "2.11.8" // Make sure to have installed Scala 11
sparkVersion := "2.0.0"  // Make sure to have installed Spark 2.0

Note that when compiling with sbt package, the .jar will now be created in target/scala-2.11/, and the .jar name will also be changed, so the spark-submit command need to be updated as well.

Update ML Vector libraries

ML Transformers now generates org.apache.spark.ml.linalg.VectorUDT instead of org.apache.spark.mllib.linalg.VectorUDT.

They are also mapped locally to subclasses of org.apache.spark.ml.linalg.Vector. These are not compatible with old MLLib API which is moving towards deprecation in Spark 2.0.0.

//import org.apache.spark.mllib.linalg.{Vector, Vectors} // Depreciated in Spark 2.0 
import org.apache.spark.ml.linalg.Vector // Use instead

Contributors

Topic Id: 6506

Example Ids: 22321,22322

This site is not affiliated with any of the contributors.