Course

Introduction to Spark : Spark Architecture, Spark Jobs and APIs. Resilient Distributed Datasets- Creating RDDs, Transformation, Actions. Dataframes- Python to RDD communications, Creating Dataframes, Dataframe queries. MLlib -Loading and Transforming the data. Implementation of Machine Learning algorithms such as Classification and Clustering using the MLlib

Approaches to Modelling- Importance of Words in Documents – Hash Functions- Indexes – Secondary Storage -The Base of Natural Logarithms – Power Laws – Map Reduce. Finding similar items: Shingling – LSH – Distance Measures. Mining Data Streams: Stream data model – Sampling data – Filtering streams. Link Analysis: Page Rank, Link Spam.

Frequent Item Sets: Market Basket Analysis, A-Priori Algorithm – PCY Algorithm, Big data Clustering: Clustering in Non-Euclidean Spaces, BFR, CURE. Structured Streaming: Spark Streaming, Application dataflow. Coresets: Coresets for K-means, K -median clustering

Preamble

This course deals with two aspects of big data analytics. The first one is the infrastructure for big data analytics. Introduction to tools and algorithms that can be used to generate models from big data and to scale those models up to big data problems. Spark framework is the chosen platform. The second is the understanding and implementation of scalable and streaming algorithms to analyze voluminous data that is growing exponentially

Course Objectives

• To understand various scalable machine learning algorithms to solve big data problems.
• To understand the SPARK architecture
• To implement Machine Learning algorithms using PySpark

Course Outcomes

Prerequisites

• Machine Learning.

CO-PO Mapping

Evaluation Pattern – 70:30

• Midterm Exam – 20%
• Lab Assignments – 25%
• Project – 25%
• End Semester Exam – 30%

Text Book / References

1. Anand Raja Raman, Jure Leskovec and J.D. Ullman, “Mining of Massive Data sets”, e-book, Publisher, 2014.
2. Kevin P. Murphey, “Machine Learning, a Probabilistic Perspective”, The MIT Press Cambridge, Massachusetts, 2012.
3. Tomasz Drabas, Denny Lee , ”Learning Pyspark”, Packt, February 2017.
4. Jeff M. Phillips, ”Coresets and Sketches”, arXiv:1601.00617,2016