Course

PROFESSIONAL ELECTIVES

Electives Electives in Data Science

Unit I

Hadoop ecosystem in Brief –Basic Paradigm and system architecture, MapRedand HDFS, Making a small Hadoop cluster –Iterative and non-Iterative batch processing, Data stores, HBASE, HIVE, PIG-New generation Big data using Functional Programming in Scala: Basic Syntax-type inference and static types-function types and value types, closures.

Unit II

Immutability and immutable types-generic type Parameters-Recursive arbitrary collections –ConsList -Iterative arbitrary collections-Arrays-Tail recursion-factorial example-functional abstractions with examples-square root, fixed point, sequence summations. Higher order functions-MapReduce Template-Pattern Matching syntax. Similar higher order (Cons) List operations on arbitrary Collections-filter, fold, partition, span. Basic entity classes and objects in Scala.

Unit III

Apache Spark: -ResilientDistributed Datasets -Creating RDDs, Lineage and Fault tolerance, DAGs, Immutability, task division and partitions, transformations and actions, lazy evolutions and optimization -Formatting and housing data from spark RDDs–Persistence. Setting up a standalone Spark cluster-: spark-shell, basic API, Modules-Core, Key/Value pairs and other RDD features, MLlib-examples for bi-class SVM and logistic regression.

Course Objectives

• To provide in-depth knowledge about big data Technologies and tools used for big data.
• To facilitate students to learn to implement and work on tools to handle large volumes of data in parallel and distributed environments.
• To throw light on retrieval and analysis of unstructured data using NOSQL databases.
• To impart in-depth knowledge of Spark and Spark MLlib.

Course Outcomes

CO1: Understand fundamental concepts of Big Data and its technologies.

CO2: Apply concepts of MapReduce framework for optimization.

CO3: Analyze appropriate NoSQL database techniques for storing and processing large volumes of structured and unstructured data.

CO4: Apply data analytics solutions using Hadoop ecosystems and Spark.

CO5: Explore modern big data processing packages for Machine learning.

CO-PO Mapping

Evaluation Pattern: 70:30

*CAT – Can be Quizzes, Assignments, and Reports

*CAL – Can be Lab Assessments, Project, and Report

**End Semester can be theory examination/ lab-based examination/ project presentation

Textbook(s)

Holden Karau, Andy Konwinski, Patrick Wendell and MateiZaharia, “Learning Spark: Lightning-Fast Big Data Analysis”, 1st Edition,2015.

Reference(s)

Cay S. Horstmann, “Scala for the Impatient”, 2nd Edition,2017.

Bill Chambers and MateiZaharia, “Spark: The Definitive Guide: Big Data Processing Made Simple”, 1st Edition,2018.

Martin Odersky, Lex Spoon and Bill Venners, “Programming in Scala: A Comprehensive Step-by-Step Guide”, 3rd Edition,2008.

Holden Karau and Rachel Warren, “High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark”, 1st Edition,2017.

Tom White, “Hadoop: The Definitive Guide”, 4th Edition, 2015.