Master Big Data - Apache Spark/Hadoop/Sqoop/Hive/Flume
- Description
- Programme
- Commentaires
In this course, you will start by learning what is hadoop distributed file system and most common hadoop commands required to work with Hadoop File system.
Then you will be introduced to Sqoop Import
-
Understand lifecycle of sqoop command.
-
Use sqoop import command to migrate data from Mysql to HDFS.
-
Use sqoop import command to migrate data from Mysql to Hive.
-
Use various file formats, compressions, file delimeter,where clause and queries while importing the data.
-
Understand split-by and boundary queries.
-
Use incremental mode to migrate the data from Mysql to HDFS.
Further, you will learn Sqoop Export to migrate data.
-
What is sqoop export
-
Using sqoop export, migrate data from HDFS to Mysql.
-
Using sqoop export, migrate data from Hive to Mysql.
Further, you will learn about Apache Flume
-
Understand Flume Architecture.
-
Using flume, Ingest data from Twitter and save to HDFS.
-
Using flume, Ingest data from netcat and save to HDFS.
-
Using flume, Ingest data from exec and show on console.
-
Describe flume interceptors and see examples of using interceptors.
-
Flume multiple agents
-
Flume Consolidation.
In the next section, we will learn about Apache Hive
-
Hive Intro
-
External & Managed Tables
-
Working with Different Files – Parquet,Avro
-
Compressions
-
Hive Analysis
-
Hive String Functions
-
Hive Date Functions
-
Partitioning
-
Bucketing
Finally You will learn about Apache Spark
-
Spark Intro
-
Cluster Overview
-
RDD
-
DAG/Stages/Tasks
-
Actions & Transformations
-
Transformation & Action Examples
-
Spark Data frames
-
Spark Data frames – working with diff File Formats & Compression
-
Dataframes API’s
-
Spark SQL
-
Dataframe Examples
-
Spark with Cassandra Integration
-
9Sqoop IntroductionVideo lesson
-
10Managing Target DirectoriesVideo lesson
-
11Working with Parquet File FormatVideo lesson
-
12Working with Avro File FormatVideo lesson
-
13Working with Different CompressionsVideo lesson
-
14Conditional ImportsVideo lesson
-
15Split-by and Boundary QueriesVideo lesson
-
16Field delimetersVideo lesson
-
17Incremental AppendsVideo lesson
-
18Sqoop-Hive Cluster FixText lesson
-
19Sqoop Hive ImportVideo lesson
-
20Sqoop List Tables/DatabaseVideo lesson
-
21Sqoop Assignment1Text lesson
-
22Sqoop Assignment2Text lesson
-
23Sqoop Import Practice1Video lesson
-
24Sqoop Import Practice2Video lesson
-
29Flume Introduction & ArchitectureVideo lesson
-
30Exec Source and Logger SinkVideo lesson
-
31Moving data from Twitter to HDFSVideo lesson
-
32Moving data from NetCat to HDFSVideo lesson
-
33Flume InterceptorsVideo lesson
-
34Flume Interceptor ExampleVideo lesson
-
35Flume Multi-Agent FlowVideo lesson
-
36Flume ConsolidationVideo lesson
-
37Hive IntroductionVideo lesson
-
38Hive DatabaseVideo lesson
-
39Hive Managed TablesVideo lesson
-
40Hive External TablesVideo lesson
-
41Hive InsertsVideo lesson
-
42Hive AnalyticsVideo lesson
-
43Working with ParquetVideo lesson
-
44Compressing ParquetVideo lesson
-
45Working with Fixed File FormatVideo lesson
-
46Alter CommandVideo lesson
-
47Hive String FunctionsVideo lesson
-
48Hive Date FunctionsVideo lesson
-
49Hive PartitioningVideo lesson
-
50Hive BucketingVideo lesson
-
56Map/FlatMap TransformationVideo lesson
-
57Filter/IntersectionVideo lesson
-
58Union/Distinct TransformationVideo lesson
-
59GroupByKey/ Group people based on Birthday monthsVideo lesson
-
60ReduceByKey / Total Number of students in each SubjectVideo lesson
-
61SortByKey / Sort students based on their rollnoVideo lesson
-
62MapPartition / MapPartitionWithIndexVideo lesson
-
63Change number of PartitionsVideo lesson
-
64Join / join email address based on customer nameVideo lesson
-
65Spark ActionsVideo lesson
-
72Dataframe IntroVideo lesson
-
73Dafaframe from Json FilesVideo lesson
-
74Dataframe from Parquet FilesVideo lesson
-
75Dataframe from CSV FilesVideo lesson
-
76Dataframe from Avro FileVideo lesson
-
77Working with XMLVideo lesson
-
78Working with ColumnsVideo lesson
-
79Working with StringVideo lesson
-
80Working with DatesVideo lesson
-
81Dataframe Filter APIVideo lesson
-
82DataFrame API Part1Video lesson
-
83DataFrame API Part2Video lesson
-
84Spark SQLVideo lesson
-
85Working with Hive Tables in SparkVideo lesson
![6542](https://academiaraqmya.gov.ma/wp-content/uploads/2021/03/2170564_8911_12.jpg)