Course Overview
What you’ll learn
Ingesting data from multiple sources. Batch Data Ingestion, Streaming Data Ingestion. Storing data in DataLake. Data Curation. Exploratory Data Analysis. Data Organization and Data Lineage.
Requirements
- Spark fundamentals
- Pyspark Fundamentals
- Hadoop Fundamentals (listed below)
- Kafka
- HBase or Hive
- Atlas
Target audiences
- Data Architects, Data Engineers, Data Scientists
Curriculum
- 1 Section
- 13 Lessons
- 3 Days
Expand all sectionsCollapse all sections
- Topics13
- 1.1Part I. Data Science Concepts Unit 1. Introduction to Data Processing Data Processing pipeline. Data Quality. Data Ingestion. Data Curation. Exploratory Data Analysis. Data Organization and Lineage. Data Processing Policies. Part II. Data Ingestion
- 1.2Unit 2. Introduction to Data Ingestion Reading data from multiple sources. Raw data and Data Curation. Batch Processing. Stream Processing using Kafka. Storing data in Data Lake.
- 1.3Unit 3. Batch Processing Architecture and abstractions. Reading data from multiple sources. Data transformations. Storing data in a Data Lake.
- 1.4Unit 4. Spark Streaming Architecture and Abstractions, Transformations, Discretized Streams (DStreams), Input Dstreams and Receivers, Transformations on DStreams Lab included
- 1.5Unit 4. Spark Streaming Architecture and Abstractions, Transformations, Discretized Streams (DStreams), Input Dstreams and Receivers, Transformations on DStreams Lab included
- 1.6Unit 6. Spark Streaming Integrations Performance Tuning, pySpark Streaming, Kafka and Structured Streaming, Kafka and DStreams. Lab Included
- 1.7Unit 7. Advanced Spark Streaming Event Time Windows, Processing time Windows, Watermarking. Lab included
- 1.8Unit 8. Advanced Structured Streaming Structured Streaming fault tolerance and recovery. Structured Streaming Performance Tuning considerations. Monitoring Structured Streaming Lab included Part III. Exploratory Data analysis
- 1.9Unit 9. Exploratory Data Analysis What is Exploratory Data Analysis (EDA). EDA steps and implementation. Lab included
- 1.10Unit 10. Exploratory Data Analysis on Structured Data Exploratory Data Analysis functions on Different types of data Lab included
- 1.11Unit 11. Exploratory Data Analysis on UnStructured Data Exploratory Data Analysis functions on Unstructured Data. Part IV. Data Organization
- 1.12Unit 12. Data Lineage Data Lineage. Using and applying data Lineage functions.
- 1.13Unit 13. Data Organization DataSet organization methodology. Data Set search.
Instructor
