{"id":5529,"date":"2021-11-08T13:23:14","date_gmt":"2021-11-08T13:23:14","guid":{"rendered":"https:\/\/www.npntraining.com\/blog\/?p=5529"},"modified":"2022-01-02T07:42:07","modified_gmt":"2022-01-02T07:42:07","slug":"various-entry-points-for-apache-spark","status":"publish","type":"post","link":"https:\/\/www.npntraining.com\/blog\/various-entry-points-for-apache-spark\/","title":{"rendered":"Various Entry Points for Apache Spark"},"content":{"rendered":"

In\u00a0Data Engineering<\/strong>\u00a0Apache Spark is probably one of the most popular framework to process huge volume of data. In this blog post I am going to cover the various entry points for Spark Applications and how these have evolved over the releases made.<\/p>\n

Every Spark Application needs an entry point that allows it to communicate with data sources and perform operations such as reading and writing data.<\/p>\n

In Spark 1.x, three entry points were introduced:<\/p>\n

    \n
  1. SparkContext,<\/li>\n
  2. SQLContext and<\/li>\n
  3. HiveContext<\/li>\n<\/ol>\n

    Since Spark 2.x, a new entry point called SparkSession has been introduced that essentially combined all functionalities available in the three aforementioned contexts. Note that all contexts are still available even in newest Spark releases, mostly for backward compatibility purposes.<\/p>\n

    Spark Context<\/h2>\n

    The Spark Context is used by the Driver Process of the Spark Application in order to establish a communication with the cluster and the resource managers in order to coordinate and execute jobs. SparkContext also enables the access to the other two contexts, namely SQLContext and HiveContext (more on these entry points later on).<\/p>\n

    In order to create a SparkContext, you will first need to create a Spark Configuration<\/p>\n

    Scala<\/em><\/strong><\/p>\n

    import org.apache.spark.{SparkContext, SparkConf}\nval sparkConf = new SparkConf().setAppName("app")\n.setMaster("yarn")\nval sc = new SparkContext(sparkConf)<\/code><\/pre>\n
    Python<\/em><\/strong><\/p>\n
    from pyspark import SparkContext, SparkConf\nconf = SparkConf().setAppName('app')\n.setMaster(master)\nsc = SparkContext(conf=conf)<\/code><\/pre>\n
    Note : if you are using the spark-shell, SparkContext is already available through the variable called sc<\/p>\n
    SqlContext<\/h2>\n
    SQLContext is the entry point to SparkSQL which is a Spark module for structured data processing. Once SQLContext is initialized, the user can then use it in order to perform various \u201csql-like\u201d operations over Datasets and Dataframes. In order to create a SQLContext, you first need to instantiate a SparkContext as shown below:<\/p>\n
    Scala<\/em><\/strong><\/p>\n
    import org.apache.spark.{SparkContext, SparkConf}\nimport org.apache.spark.sql.SQLContext\nval sparkConf = new SparkConf().setAppName("app")\n.setMaster("yarn")\nval sc = new SparkContext(sparkConf)\nval sqlContext = new SQLContext(sc)<\/code><\/pre>\n
    Python<\/em><\/strong><\/p>\n
    from pyspark import SparkContext, SparkConf\nfrom pyspark.sql import SQLContext\nconf = SparkConf().setAppName('app')\n.setMaster(master)\nsc = SparkContext(conf=conf)\nsql_context = SQLContext(sc)<\/code><\/pre>\n
    HiveContext<\/h2>\n
    If your Spark Application needs to communicate with Hive and you are using Spark < 2.0 then you will probably need a HiveContext if . For Spark 1.5+, HiveContext also offers support for window functions.<\/p>\n
    Scala<\/em><\/strong><\/p>\n
    import org.apache.spark.{SparkConf, SparkContext} \nimport org.apache.spark.sql.HiveContext\nval sparkConf = new SparkConf().setAppName(\"app\")\n                               .setMaster(\"yarn\")\nval sc = new SparkContext(sparkConf)\nval hiveContext = new HiveContext(sc)\nhiveContext.sql(\"select * from tableName limit 0\")\n<\/code><\/pre>\n
    Python<\/em><\/strong><\/p>\n
    from pyspark import SparkContext, \nHiveContextconf = SparkConf().setAppName('app')\n                             .setMaster(master)\nsc = SparkContext(conf)\nhive_context = HiveContext(sc)\nhive_context.sql(\"select * from tableName limit 0\")<\/code><\/pre>\n
    SparkSession<\/h2>\n
    Spark 2.0 introduced a new entry point called SparkSession that essentially replaced both SQLContext and HiveContext. Additionally, it gives to developers immediate access to SparkContext. In order to create a SparkSession with Hive support, all you have to do is<\/p>\n
    import org.apache.spark.sql.SparkSession\nval sparkSession = SparkSession.builder()\n                                .appName(\"myApp\")\n                                .enableHiveSupport()\n                                .getOrCreate()<\/code><\/pre>\n
    \/\/ Two ways you can access spark context from spark session<\/p>\n
    val spark_context = sparkSession._sc\nval spark_context = sparkSession.sparkContex<\/code><\/pre>\n
    Python<\/em><\/strong><\/p>\n
    from pyspark.sql import SparkSession\nspark_session = SparkSession.builder.enableHiveSupport().getOrCreate()<\/code><\/pre>\n
    Two ways you can access spark context from spark session<\/p>\n
    spark_context = spark_session._sc\nspark_context = spark_session.sparkContex<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
    In\u00a0Data Engineering\u00a0Apache Spark is probably one of the most popular framework to process huge volume of data. In this blog post I am going to cover the various entry points…<\/p>\n","protected":false},"author":1,"featured_media":5712,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[54,50],"tags":[],"class_list":["post-5529","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","category-data-engineering"],"_links":{"self":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/comments?post=5529"}],"version-history":[{"count":24,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5529\/revisions"}],"predecessor-version":[{"id":7862,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5529\/revisions\/7862"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/media\/5712"}],"wp:attachment":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/media?parent=5529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/categories?post=5529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/tags?post=5529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}