Overview

This four day course of Spark Developer is for data engineers,  analysts, architects; software engineers; IT operations; and technical managers interested in a thorough, hands-on overview of Apache Spark.

 The course covers the core APIs for using Spark, fundamental mechanisms and basic internals of the framework, SQL and other high-level data access tools, as well as Spark’s streaming capabilities and machine learning APIs.

 Objectives

After taking this class you will be able to:

• Describe Spark’s fundamental mechanics
• Use the core Spark APIs to operate on data
• Articulate and implement typical use cases for Spark
• Build data pipelines with SparkSQL and DataFrames
• Analyze Spark jobs using the UIs and logs Create Streaming and Machine Learning jobs

Pre requisitie : 

• Required

◦               Basic to intermediate Linux knowledge, including: The ability to use a text editor, such as vi

Familiarity with basic command-line options such a mv, cp, ssh, grep, cd, useradd

◦                            Knowledge of application development principles

•   Recommended

◦                            Knowledge of functional programming

◦                            Knowledge of Scala or Python

◦                            Beginner fluency with SQL

 Course Overview

Lesson 1 – Introduction to Apache Spark 

• Describe the features of Apache Spark
• Advantages of Spark
• How Spark fits in with the Big Data application stack
• How Spark fits in with Hadoop
• Define Apache Spark components 

Lesson 2 – Load and Inspect Data in Apache Spark 

• Describe different ways of getting data into Spark
• Create and use Resilient Distributed Datasets (RDDs)
• Apply transformation to RDDs
• Use actions on RDDs
• Load and inspect data in RDD
• Cache intermediate RDDs
• Use Spark DataFrames for simple queries
• Load and inspect data in DataFrames

Lesson 3 – Build a Simple Apache Spark Application

• Define the lifecycle of a Spark program
• Define the function of SparkContext
• Create the application
• Define different ways to run a Spark application
• Run your Spark application
• Launch the application

Lesson 4 – Work with PairRDD 

• Review loading and exploring data in RDD
• Load and explore data in RDD
• Describe and create Pair RDD
• Create and explore PairRDD
• Control partitioning across nodes

Lesson 5 – Work with DataFrames 

• Create DataFrames

◦                            From  existing RDD

◦                            From  data sources

• Work with data in DataFrames

◦                            Use DataFrame operations

◦                            Use SQL

◦                            Explore data in DataFrames

• Create user-defined functions (UDF)

◦                            UDF used with Scala DSL

◦                            UDF used with SQL

◦                            Create and use user-defined functions

• Repartition DataFrames
• Supplemental Lab: Build a standalone application

Lesson 6 – Monitor Apache Spark Applications 

• Describe components of the Spark execution model
• Use Spark Web UI to monitor Spark applications
• Debug and tune Spark applications
• Use the Spark Web U 

Lesson 7 – Introduction to Apache Spark Data Pipelines

Identify components of Apache Spark Unified Stack

• List benefits of Apache Spark over Hadoop ecosystem
• Describe data pipeline use cases

Lesson 8 – Create an Apache Spark Streaming Application