In this blog post we will learn how to convert RDD to DataFrame with spark helper methods used in local development or testing.<\/p>\n
First let us create an RDD from collections,<\/p>\n
val temperatureRecords = Seq(\n(India,Array(27.0,26.0,40.1)),\n(China,Array(27.0,26.0,40.1))\n)\nval temperatureRDD = spark.sparkContext.parallize(temperatureRDD);<\/pre>\nThere are several ways to convert RDD to DataFrame<\/p>\n
\n
- By using createDataFrame(RDD obj) from SparkSession object<\/li>\n
- By using\u00a0createDataFrame(RDD obj) from SparkSession object and by specifying columns names<\/li>\n
- By using createDataFrame(RDD obj, StructType type) by providing schema using StructType<\/li>\n<\/ol>\n
Method 01 -\u00a0<\/strong><\/h2>\n
We will use createDataFrame(Rdd rdd) method to convert RDD into DataFrame<\/p>\n
Creating RDD with collections and converting into DataFrame<\/p>\n
val temperatureRecords = List(("Afghanistan","8,996,351"),("China","667,070.00"),("India","449,480.61"))\nval dfWithoutSchema = spark.createDataFrame(temperatureData)\ndfWithoutSchema.show;<\/pre>\nOutput<\/strong><\/p>\n
+-----------+----------+\n| _1| _2|\n+-----------+----------+\n|Afghanistan| 8,996,351|\n| China|667,070.00|\n| India|449,480.61|\n+-----------+----------+<\/pre>\nCreating RDD with external source and converting into DataFrame<\/p>\n
val temperatureData = spark.sparkContext.textFile("d:\/spark-example\/population.csv");<\/pre>\nOutput<\/strong><\/p>\n
+-----------+----------+\n| _1| _2|\n+-----------+----------+\n|Afghanistan| 8,996,351|\n| China|667,070.00|\n| India449| 480.61|\n+-----------+----------+<\/pre>\nIf you notice the above output the columns names are "_1" and "2" since we didn't specify the column names. Lets see how to define column names in the next method.<\/p>\n
Method 02 -\u00a0<\/strong><\/h2>\n
toDF() provides a concise syntax for creating DataFrames\u00a0 by specifying column names and can be accessed after importing Spark implicits.<\/p>\n
import sqlContext.implicits.<\/em><\/pre>\nval dataFrameWithSchema = spark.createDataFrame(temperatureRecords).toDF("Country","Population in billions");\ndataFrameWithSchema.show;<\/pre>\nOutput<\/strong><\/p>\n
+-----------+----------------------+\n| Country|Population in billions|\n+-----------+----------------------+\n|Afghanistan| 8,996,351|\n| China| 667,070.00|\n| India449| 480.61|\n+-----------+----------------------+<\/pre>\nMethod 03 -\u00a0<\/strong><\/h2>\n
In this method we will see how to provide schema by using StructType<\/p>\n
val schema = new StructType()\n.add(StructField("id", StringType, true))\n.add(StructField("population", LongType, true))\n)\nva dataFrame = spark.createDataFrame(rowsRDD,schema)\ndataFrame.show;<\/pre>\n