Apache Spark Introduction

What is Apache Spark

Apache Spark is a cluster computing platform designed to be fast and general purpose. On the speed side, Spark extends the popular MapReduce model to efficiently support more types of computations, including interactive queries and stream processing.

A Unified Stack

The Spark project contains multiple closely integrated components. At its core, Spark is a “computational engine” that is responsible for scheduling, distributing, and monitoring applications consisting of many computational tasks across many worker machines, or a computing cluster. Because the core engine of Spark is both fast and general-purpose, it powers multiple higher-level components specialized for various workloads, such as SQL or machine learning.

We will briefly introduce each of Spark’s components,

Spark Core

  1. Spark Core contains the basic functionality of Spark, including components for task scheduling, memory management, fault recovery, interacting with storage systems, and more.
  2. Spark Core is also home to the API that defines resilient distributed datasets (RDDs), which are Sparks’s main programming abstraction.

Spark SQL

  1. Spark SQL is Spark’s package for working with structured data. It allows querying data via SQL as well as the Apache Hive variant of SQL—called the Hive Query Language (HQL)— and it supports many sources of data, including Hive tables, Parquet, and JSON.
  2. Spark SQL allows developers to intermix SQL queries with the programmatic data manipulations supported by RDDs in Python, Java, and Scala, all within a single application, thus combining SQL with complex analytics.

Spark Streaming

  1. Spark Streaming is a Spark component that enables processing of live streams of data. Examples of data streams include logfiles generated by production web servers, or queues of messages containing status updates posted by users of a web service.


  1. Spark comes with a library containing common machine learning (ML) functionality, called MLlib.
  2. MLlib provides multiple types of machine learning algorithms, including classification, regression, clustering, and collaborative filtering, as well as supporting functionality such as model evaluation and data import.


  1. GraphX is a library for manipulating graphs (e.g., a social network’s friend graph) and performing graph-parallel computations.