Spark and the RDD (Resilient Distributed Dataset)

Fast, robust, general-purpose cluster computing for data science

Neal McBurnett, 2015-04-09, Boulder Linux Users Group

  • Audience: BLUG: Linux users and sysadmins interested in tools and infrastructure for data science

  • For extra geek credit: live IPython notebook slides that run in a Docker image

  • Huge thanks for

Outline

  • Landscape of Distributed Computing, Trends
  • How Spark fits in
  • Resilient distributed datasets (RDDs)
  • Functional programming and compiling for your cluster
  • Spark's programming model
  • Dataframes in Spark
  • Deployment environments
  • Links to explore

Landscape of Distributed Computing

Many-Task Computing: Bridging the Gap between High Throughput Computing and High Performance Computing

  • Cheap network => Reference: URI
  • Cheap storage => Replicate: Distributed File System
  • Cheap memory => Recompute: Resilient Distributed Dataset

Data Storage

Hadoop Definitive Guide by Thom White

  • Access speeds are not keeping up with capacity
  • HDFS motivation: e.g 100 MB/s

MapReduce

MapReduce: Simplified Data Processing on Large Clusters

Kernel programming model for working with distributed data.

  • Inefficiencies with MapReduce - each iteration reads and writes data to disk
  • Storage costs dropping fast, but transfer rates not keeping up
  • What if we just keep it all in memory? Prone to failure.

Spark

Spark

  • UC Berkeley AMPLab, 2009....
  • One of the largest Open Source communities in big data
  • Over 250+ contributors in 50+ organizations
  • Goals: general, low-latency, fault-tolerant, simple
  • Spark the fastest open source engine for sorting a petabyte | Databricks
    • Daytona Gray industry benchmark: sort 100 TB of data, 1 trillion records
    • Spark sorted the same data 3X faster than previous world record (Hadoop) using 10X fewer machines
    • Up to 100x faster than Hadoop when in-memory caching is effective

Spark officially sets 2014 record in large-scale sorting | Databricks

Use cases

  • Data science
  • Log processing
  • Machine learning
  • Full-text search of wikipedia in < 1 second on 20-node cluster vs 30 seconds from disk
  • Popular due to speed, flexibility, blend of interactive investigative analytics and real-time enterprise data analytics that can drive business processes
  • Batch operation or leverage streaming data on the fly
  • Straightforward enough for everyday use to parallelize work you might do in R or Pandas

Functional programming, Lazy evaluation for Clusters

  • Lucky to learn some functional programming back in 1970's
  • Much more mature and important now!
  • Functional programming style pretty well established in modern languages
  • Parallel processing critical for big workloads
  • Lack of side effects in functional programming enormously useful in parallel processing
  • Lazy evaluation helps greatly with latency
  • Spark mashes this up with SQL optimizing technology: Catalyst compiler for big computations to your cluster
  • Paco Nathan, Just Enough Math - O'Reilly Media
    • With links to over an hour of free video on functional programming and monoids for cluster computation

RDD (Resilient Distributed Dataset)

  • New abstraction that led to Spark
  • Avoid data replication for fault tolerance
  • Instead define collection dataset, partitioned across cluster
    • Definition includes lineage - how each RDD depends on others
    • In general, a Directed Acyclic Graph (DAG)
    • Immutable: if you want to change dataset, store in a new RDD
    • exceptions: can do mutable RDD for Spark streaming or Cassandra
  • RDDs offer an API based on coarse-grained transformations that lets them recover data efficiently using lineage.
  • Two types of RDD operations
    • Actions: return values (e.g. count)
    • Transformations: define new RDDs based on extisting ones
      • filter; map; reduce

Example of how Spark computes job stages. Boxes with solid outlines are RDDs. Partitions are shaded rectangles, in black if they are already in memory

Spark: it slices, it dices....

Deployment Environments

  • Kinds of clusters; Mesos (general purpose), YARN (hadoop)
  • Kinds of file systems
    • HDFS - Hadoop File System - widely used in enterprises
    • Cassandra: lighter-weight distributed filesystem for structured data - convenient integration with Spark 1.3 DataFrame API
    • Tachyon: in-memory, from Spark
  • Language support: Scala, Java, Python, SQL, R (coming)

DataFrame API

  • New in Spark 1.3
  • An evolution of Spark’s RDD abstraction designed to make crunching large datasets simple and fast
  • Inspired by, and fully interoperable with, Pandas and R data frames
  • Internally leverage the Spark SQL logical optimizer. Intelligently plan the physical execution of operations to work well on large datasets. This planning permeates all the way into physical storage, where optimizations such as predicate pushdown are applied based on analysis of user programs.
  • Create directly from Hive tables, Parquet files, Spark SQL, etc.

Practicalities

  • Most of the runtime and libraries are Scala or Java, cleverly integrated with Python
  • Lots of java stuff running behind the scenes
  • With hadoop, lots more processes too
  • Many hassles with modern IPython stuff (and Python 2.7) if you've got CentOS/RHEL 6
  • Not yet officially supported on python 3, but getting close?

Explore more

Questions?

In [2]:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
In [21]:
%%bash
hdfs dfs -rm -r -f ints.parquet  # Delete ints.parquet if it was created previously
In [22]:
import random
from pyspark.sql import Row

data = sc.parallelize(xrange(100)).flatMap(lambda x: [Row(a=random.randint(1, 10), num=random.randint(1, 100), str=("a" * random.randint(1, 30))) for i in xrange(1000)])
dataTable = sqlContext.createDataFrame(data)
dataTable.saveAsParquetFile("ints.parquet")
In [23]:
data
Out[23]:
PythonRDD[34] at RDD at PythonRDD.scala:42
In [12]:
data = sqlContext.load("ints.parquet")
data.groupBy("a").agg(col("a"), avg("num")).collect()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-12-be2cb65442b5> in <module>()
      1 data = sqlContext.load("ints.parquet")
----> 2 data.groupBy("a").agg(col("a"), avg("num")).collect()

NameError: name 'col' is not defined