Introduction to Big Data

1. 課程資料 Course outline & logistics

第一堂前半部分係課程行政資料 (slides 2–13)。雖然唔係「知識」，但 mid-term 範圍、評分比重等資訊對你規劃溫習好重要。

• Instructor：Dr. Aileen Hou (Lecture + Lab)，Department of Computer Science，Office N407。Consultation 可以 email、預約 face-to-face / online meeting，或者 MS Teams PM。
• Module nature：Foundation course — Python basics 同 Python for data processing and analysis。即係話本科假設你由零開始學 Python，之後用 Python (pandas) 做 data processing。
• References：Skiena The Data Science Design Manual (2017)；McKinney Python for Data Analysis 2nd ed. (2017)；EMC Data Science and Big Data Analytics (2015)；MIT OCW Linear Algebra (18.06)。另外 Moodle 有 vocabulary list。

Teaching schedule (tentative)

Week	Lecture	Week	Lecture
1	Introduction	8	EDA 1 (+ Test)
2	Python crash course 1	9	EDA 2
3	Python crash course 2	10	Data visualization
4	Data manipulation	11	Big data eco system and ethics
5	Data cleaning 1	12	Revision and Q&A
6	Data cleaning 2	13	Presentation
7	Feature engineering & project briefing

Assessments

Component	Weight	重點
Class Participation	10%	Weekly lab exercises + attendance；評分睇你有冇「put effort to do and learn」，答案啱唔啱未必計分。
Group Assignment	30%	2–3 人一組 (L01 & L02 可混合，最多 24 組)；Python data analysis project；一個 Notebook 入面要有 code、plots、descriptions；要做 oral presentation。
Mid-term Test	20%	24 Oct 19:00 (L01 & L02 一齊)；Coverage: lectures and labs in weeks 1–7。
Final Examination	40%	Week 14–16 考試期。

Weekly lab exercises 嘅任務係「given a dataset, use Python to perform: data loading → preparation → cleaning → analysis → visualization」— 呢條 pipeline 其實就係成科嘅主線，亦都對應後面講嘅 CRISP-DM。

Class policy / Tips for success：70%+ attendance；請假要上堂前 email；嚴禁 plagiarism (VeriGuide 會查)，用 generative AI 要跟大學同學系指引；同一份功課唔可以喺唔同科交；group work 唔准 free rider；遲交會扣分；盡量唔好靠翻譯軟件。Slide 13 「How to survive in HSU?」只係一個標題頁。

2. What is Data? Objects & Attributes

Slide 15 先定義 structured data：一堆 objects，每個 object 用一組 attributes / features 嚟描述。可以想像成一個表格：

• 每一 row = 一個 object (又叫 record, point, case, sample, entity, instance)。
• 每一 column = 一個 attribute / feature。

Slide 用咗兩個例子：

Book No.	Title	Author		Student ID	Height (cm)	Weight (kg)
001	Data Mining	John		001	175	62
002	Robots	Edward		002	179	65
003	Computer	Dan		003	168	60
004	Mathematics	Mary		004	181	59

左邊每本書係一個 object，attributes 係 Book No. / Title / Author；右邊每個學生係 object，attributes 係 Student ID / Height / Weight。

import pandas as pd
# 用 slide 15 右邊個例子建一個 DataFrame：每 row 係一個 object，每 column 係一個 attribute
students = pd.DataFrame({
    "Student ID": ["001", "002", "003", "004"],
    "Height (cm)": [175, 179, 168, 181],
    "Weight (kg)": [62, 65, 60, 59],
})
print(students.shape)   # (4, 3) → 4 個 objects，3 個 attributes

3. Types of Attributes

Exam Slide 16 把 attributes 分成兩大類，每類再細分：

大類	細分	特點	Slide 例子
Categorical	Nominal	只係名稱 / 標籤，no order or comparison	eye color, zip codes, words
	Ordinal	有次序 (order)，但兩級之間嘅「距離」not comparable (唔可以計差幾多)	rankings (good, fair, bad)、height in {tall, medium, short}
Numerical	Discrete	可數嘅值 (counts)	counts
	Continuous	可以喺一個範圍內取任何實數值	temperature, length, height in cm, weight in kg
Special case: Binary		只有兩個值	yes/no, exists/not exists

import pandas as pd
df = pd.DataFrame({
    "eye_color": ["blue", "brown", "green"],          # Nominal
    "rating": ["good", "bad", "fair"],                # Ordinal
    "n_children": [0, 2, 1],                          # Discrete
    "temperature": [36.5, 37.2, 36.8],                # Continuous
    "smoker": [True, False, False],                   # Binary
})
# Ordinal 可以用 ordered categorical 表示次序
df["rating"] = pd.Categorical(df["rating"], categories=["bad", "fair", "good"], ordered=True)
print(df.dtypes)
print(df["rating"] > "fair")   # 有次序先可以比較大細

4. What is Data Analytics?

Slide 17：Data Analytics = Knowledge discovery from data! Data 本身蘊含 value 同 knowledge，analytics 就係把佢挖出嚟。

一般嚟講 DIKW 指 Data → Information → Knowledge → Wisdom 由下而上嘅金字塔：raw data (例如一堆 sales 數字) 經整理變成 information (「上個月 A 店賣咗 500 件」)，再歸納成 knowledge (「下雨天 A 店銷量會跌」)，最後變成可以指導行動嘅 wisdom (「天氣預報落雨就減少 A 店入貨」)。

Exam Slide 18：Given lots of data，我哋想發現嘅 patterns / models 要符合四個條件：

條件	Slide 定義	解釋
Valid	hold on new data with some certainty	唔係只喺手頭 data 成立 (唔係 overfitting / 巧合)，放落新 data 都大致成立。
Useful	should be possible to act on the item (e.g., making decisions)	要可以據此採取行動。
Unexpected	non-obvious to the system	如果係人人都知嘅嘢 (「買麵包嘅人都會買嘢」) 就冇價值。
Understandable	humans should be able to interpret the pattern	人要睇得明、解釋得到。

Slide 用咗經典例子 “Diaper and Beer!”：超市發現買尿片 (diaper) 嘅顧客好多時都會買啤酒 (傳說係爸爸被叫去買尿片，順手買啤酒)。呢個 pattern 係 unexpected (直覺上兩者冇關)、useful (可以將兩樣貨擺近啲 / 一齊做推廣)、understandable (解釋得到)、亦需要 valid (喺新交易都成立)。

5. Where does data come from?

Slide 19：data 「coming from everywhere」：posts to social media sites、digital pictures and videos、software logs、cameras、microphones、sensing data、scans of government documents、GPS trails、purchase transaction records、cell phone GPS signals、traffic 等等。重點係：data 來源極多樣，有 text、image、video、sensor stream、交易紀錄 — 呢個就引出下一節嘅 Big Data 特徵 (尤其 Variety 同 Velocity)。

6. Characteristics of Big Data: the 5 V's

Exam Slide 20 列出 Big Data 嘅 5 V's：

V	Slide 細項	意思 & 例子
Volume	—	數據量巨大 (TB、PB 級)，單一部機存唔落 / 處理唔到。例：Facebook 每日上載嘅相。
Velocity	Incoming Stream；Real-time Analytic	數據產生同需要處理嘅速度好快，係連續流入嘅 stream，需要 real-time analytics。例：股票報價、信用卡 fraud detection。
Variety	Social Network (Graph)；Tweet Message (Text, Image, Video)；Click Log (Relational)	數據格式多樣：graph、text、image、video、relational table 等。
Veracity	Accuracy	數據嘅準確性 / 可信度：有 noise、錯誤、missing values、假資料。
Value	—	最終要由數據提煉出商業 / 社會價值，否則收集再多都冇用。

7. Data Analytic Tasks: Descriptive vs Predictive

Exam Slide 21 將 analytic tasks 分兩類：

	Descriptive methods	Predictive methods
定義 (slide)	Find human-interpretable patterns that describe the data	Use some variables to predict unknown or future values of other variables
Method 例子	Clustering, Classification	Recommendation systems (Amazon, Google Play Store)
Application 例子	Medical diagnose (use Classification)	Customer relationship management (CRM) systems
Case study	John Snow cholera map (slide 22)；Gapminder (slide 23)	Google Flu Trends (slide 24)

7.1 Clustering example: John Snow & Cholera (slide 22)

Clustering = Group similar data together. 十九世紀倫敦爆發霍亂 (Cholera)，醫生 John Snow 將每個病例畫喺城市地圖上，發現病例圍繞住某幾個路口聚集 (cluster) — 呢啲路口就係受污染嘅水井位置。住近污染水井嘅人病咗，住近乾淨水井嘅人冇事。「Without the ability to cluster the data, the cause of Cholera would not have been discovered.」

7.2 Descriptive example: Gapminder (slide 23)

Slide 23 用 Gapminder tools 嘅 bubble chart 作為 descriptive analytics 例子 — 用視覺化方式描述各國數據 (例如收入、壽命、人口) 嘅關係。

7.3 Predictive example: Google Flu Trends (slide 24)

Google 用 search history database 入面 flu-related keywords 嘅搜尋頻率，去 predict 有幾多人患流感。即係用一個 variable (搜尋量) 去預測另一個未知 / 未來嘅 variable (流感人數) — 完全符合 predictive 嘅定義。

8. The CRISP-DM Process & Data Science Processes

Exam Slide 25：CRISP-DM (CRoss Industry Standard Process for Data Mining) 係一個 process model，作為 data science process 嘅基礎。六個步驟：

1. Business Understanding — 釐清商業目標同問題 (要解決咩？成功點樣衡量？)。
2. Data Understanding — 收集同初步探索 data (有咩 attributes？質素點？)。
3. Prepare Data — cleaning、轉換、合併、feature engineering (本科 Week 4–7 重點)。
4. Data Modeling — 揀同訓練 model (clustering、classification、regression…)。
5. Evaluate the Results — 睇 model 結果係咪達到 business objectives。
6. Deploy — 將結果 / model 投入實際使用。

import pandas as pd
# 一個極簡 CRISP-DM 流程示意 (假設有 sales.csv)
# 1. Business Understanding: 想知邊類產品銷量最好
# 2. Data Understanding
df = pd.read_csv("sales.csv")
print(df.info())
# 3. Prepare Data
df = df.dropna(subset=["category", "amount"])
# 4. Modeling (此處只用簡單 aggregation 代替)
result = df.groupby("category")["amount"].sum().sort_values(ascending=False)
# 5. Evaluate: 結果有冇回答到問題？
print(result.head())
# 6. Deploy: 例如輸出報告
result.to_csv("top_categories.csv")

9. Summary (slide 27)

• What is data — attributes, data types
• What is big data — 5 V's
• Data analytics — descriptive, predictive
• The data science process — CRISP-DM

自我測驗 Self-test

Q1. 一個 object 喺 structured data 入面仲有咩其他叫法？Attributes 又叫咩？

Object 又叫 record, point, case, sample, entity, instance；attributes 又叫 features。喺表格中 object = row，attribute = column。

Q2. 將以下 attributes 分類：(a) zip code (b) customer satisfaction {low, medium, high} (c) number of purchases (d) weight in kg (e) has_membership {yes, no}

(a) Categorical – Nominal (數字但冇次序 / 計算意義)；(b) Categorical – Ordinal (有次序，但距離不可比較)；(c) Numerical – Discrete (counts)；(d) Numerical – Continuous；(e) Binary (special case)。

Q3. Nominal 同 Ordinal 有咩分別？各舉一例。

Nominal：只係名稱，no order or comparison，例如 eye color。Ordinal：有次序，但各級之間嘅距離 not comparable，例如 rankings (good, fair, bad)。

Q4. 列出 Big Data 嘅 5 V's，並指出「一間銀行要喺 0.1 秒內判斷每宗信用卡交易是否 fraud」主要係邊個 V？

Volume, Velocity, Variety, Veracity, Value。信用卡 fraud detection 要處理不斷流入嘅 transaction stream 並 real-time 分析 → 主要係 Velocity。

Q5. Data analytics 發現嘅 pattern 要符合邊四個條件？用 “Diaper and Beer” 解釋其中兩個。

Valid, Useful, Unexpected, Understandable。Unexpected：尿片同啤酒表面無關，係 non-obvious 嘅發現；Useful：超市可以將兩樣貨擺近 / 做 bundle promotion，即可據此做決策 (act on it)。

Q6. Descriptive 同 Predictive methods 嘅分別？John Snow 嘅 cholera map 同 Google Flu Trends 各屬邊類？

Descriptive：find human-interpretable patterns that describe the data。Predictive：use some variables to predict unknown or future values of other variables。John Snow → Descriptive (clustering 病例位置找出污染水井)；Google Flu Trends → Predictive (用 flu-related 搜尋頻率預測流感人數)。

Q7. 按次序寫出 CRISP-DM 六個步驟。Data cleaning 屬於邊一步？

1 Business Understanding → 2 Data Understanding → 3 Prepare Data → 4 Data Modeling → 5 Evaluate the Results → 6 Deploy。Data cleaning 屬於第 3 步 Prepare Data。

Q8. Mid-term test 嘅範圍同比重？

Lectures and labs in Weeks 1–7，佔總分 20%。(Final exam 40%，group assignment 30%，class participation 10%。)

一頁總結 Cheat sheet