Big Data in Machine Learning

• HV học qua lớp Machine Learning with Python hoặc có kiến thức tương đương
• Sinh viên các trường Đại học, Cao đẳng
• HV có định hướng sẽ làm việc trong lĩnh vực Machine Learning hoặc Data Science

• Thời gian học: 5 tuần
• Thời lượng: 64 tiết, học trực tiếp trên máy
• Học phí:  5.800.000 đ

• Overview of Big Data
  • What is Big Data?
  • History of Big Data
  • The Vs’ of Big Data (3Vs’, 4Vs’, 5Vs’)
  • Batch processing vs Stream processing
  • Introduction to Apache Spark
  • Apache Spark Components: Spark RDD API, Spark SQL, Spark MLlib, Spark GraphX, Spark Streaming
• Overview of PySpark
  • Introduction to PySpark: Spark with Python (Python API)
  • Why PySpark?
  • Installing and configuration PySpark
  • Spark context, Spark Session
• PySpark RDDs
  • Introduction to PySpark RDDs (Resilient Distributed Dataset)
  • RDDs operations
    1. Transformation
    2. Action
  • Working with PySpark RDDs
    1. Create RDD: parallelize(), textFile()
    2. RDD Transformations: map(), filter(), flatMap(), RDD1.union(RDD2)
    3. RDD Actions: collect(), take(), count(), first(), reduce(), saveAsTextFile(),…
    4. Pair RDDs:
       1. Create Pair RDDs from key-value tuple/ regular RDD
       2. Transformations: reduceByKey(), groupByKey(), sortByKey(), join()
       3. Actions: countByKey(), collectAsMap()
• PySpark DataFrame
  • Introduction to PySpark DataFrame
  • Features and Advantages
  • Working with PySpark DataFrame
    1. Create DataFrame: createDataFrame(), spark.read.csv(), spark.read.json()
    2. printSchema(), show()
    3. count()
    4. describe()
    5. crosstab()
    6. groupby()
    7. select(), select() và agg, count, max, mean, min, sum..., select().distinct(),
    8. orderby().asc()/desc()
    9. withColumn(), withColumnRenamed()
    10. drop(), dropDuplicates(), dropna()
    11. filter(), where()
    12. Column string transformation
    13. Conditional clauses: .when(<if condition>, <then x>), .otherwise()
    14. User defined functions (UDF)
  • Data Visualization in PySpark using DataFrames
    1. hist(), distplot()
    2. pandas_histogram()
• PySpark SQL
  • Introduction to PySpark SQL
  • Running SQL Queries Programmatically
    1. select()
    2. when()
    3. like()
    4. startswith(), endswith()
    5. substr(), between()
  • Manipulating data
    1. Group by
    2. Filtering
    3. Sorting
    4. Missing and replacing value
    5. Joining Data
    6. Repartitioning
    7. Registering DataFrames as Views
• Data Preprocessing & Analysis
  • Wrangling with Spark Functions
    1. Dropping, Filtering, Joining
    2. Working with missing data
    3. Using lazy processing
    4. Parquet
    5. Removing, Splitting rows/columns
    6. Data validation
  • Feature Engineering
    1. Feature Generation
    2. Differences, Ratios
    3. Deeper Features, Time Features
    4. Time Components, Joining On Time Components
    5. Date Math
    6. Extracting Features/ Text to New Features
    7. Splitting & Exploding
    8. Scaling data
    9. Pivoting & Joining
    10. Binarizing, Bucketing & Encoding
  • Data Analysis
    1. Exploratory Data Analysis (EDA), Corr
    2. Visualization: distplot, implot…
• Overview of PySpark MLlib
  • Introduction to PySpark MLlib
  • PySpark MLlib algorithms
  • Building a Model
  • Estimator and evaluator
  • Cross-validation, Grid Search
  • Interpreting Results
• Machine Learning with PySpark MLlib
  • Supervised Learning (Classification & Regression)
    1. Linear Regression (pyspark.ml.regression)
    2. Logistic Regression (pyspark.mllib.classification)
    3. Decision Tree (pyspark.mllib.classification)
    4. Random forest (pyspark.mllib.classification)
    5. Gradient-Boosted Tree
  • Pipeline
    1. Introduction to Pipeline
    2. Working with Pipeline (pyspark.ml import Pipeline)
  • Unsupervised Learning (Clustering & Recommender System)
    1. Clustering with KMeans
    2. Recommender System - ALS
    3. Association rules – FPGrowth (pyspark.ml.fpm.FPGrowth)
• PySpark Streaming
  • Introduction to PySpark Streaming
  • Why PySpark Streaming?
  • Features and Advantages
  • Streaming Context
  • DStream
  • Streaming Transformation Operations
  • Streaming Checkpoint
• Natural Language Processing - NLP
  • Tools for NLP
    1. Tokenizer
    2. StopWordsRemover
    3. NGram
    4. CountVectorizer
    5. TF-IDF
• Apache Spark standalone cluster
  • Running Master Server
  • Connecting from Slave computers to Master Server
  • Deployment project in Mater – Slave computers system
• GraphX
  • Introduction to GraphX
  • Working with GraphX
    1. Creating graph
    2. Vertex and edge
    3. Visualization Graph
    4. Filtering
    5. Connecting
    6. Motif finding
    7. Triangle count
    8. Page rank

Học viên thi đạt kết quả cuối môn học sẽ được cấp chứng nhận "Big Data in Machine Learning - Dữ liệu lớn trong Máy học" do Trung Tâm Tin Học - Trường ĐH Khoa Học Tự Nhiên cấp.