Introduction to Machine Learning: Basics of Probability Theory

Objective of the Tutorial

• High-level overview of fundamental concepts in probability theory.
• Designed as a refresher for those familiar with probability concepts.
• Recommended for those less familiar to study introductory materials prior to course content.

Key Concepts in Probability

Sample Space

• Definition: Set of all possible outcomes of an experiment.
  • Denoted by: Ω (capital omega)
  • Individual outcomes: ω (small omega)

Examples of Sample Space:

1. Rolling a Die

   • Sample space: {1, 2, 3, 4, 5, 6}
   • Finite sample space.

2. Tossing a Coin Until Condition Observed

   • Condition: Five consecutive heads.
   • Sample space: Countably infinite (sequences of H's and T's).

3. Measuring Speed of a Vehicle

   • Sample space: All real numbers (can be negative).
   • Uncountable sample space.

Events

• Definition: Any collection or subset of outcomes from the sample space.
• Example: Rolling a die, interested in whether the outcome is even or odd.
• Basic set theory notations:
  • A ⊆ B if every element of A is in B (Subset relation).
  • A = B if both A ⊆ B and B ⊆ A hold.

Set Operations

• Union: A ∪ B contains elements in either A or B.
• Intersection: A ∩ B contains only common elements.
• Complement: A' includes all elements in the universal set except those in A.

Properties of Set Operations

• Commutativity, Associativity, Distributivity.
• De Morgan's Laws:
  • (A ∪ B)' = A' ∩ B'
  • (A ∩ B)' = A' ∪ B'

Disjoint Events

• Events A and B are disjoint if A ∩ B = ∅.
• Pairwise disjoint: A1, A2, A3, ... if Ai ∩ Aj = ∅ for all i ≠ j.
• Events form a Partition of the sample space if their union covers the sample space.

Sigma Algebra

• Definition: A collection F of subsets of Ω with these properties:
  • Null set is an element of F.
  • If A ∈ F, then A' ∈ F.
  • If A_i ∈ F for all i, then ∪ A_i ∈ F.
• Events in sigma algebra are F-measurable.
• Example of sigma algebras using Ω = {1, 2, 3}.

Probability Measure and Probability Space

• A probability measure P on a sample space Ω with sigma algebra F is a function P: F → [0, 1].
• Properties:
  • P(∅) = 0
  • P(Ω) = 1
  • For pairwise disjoint A1, A2, ..., P( ∪ Ai) = ∑ P(Ai).
• The triple (Ω, F, P) forms a Probability Space.

Example: Rolling a Die

• Sample space: {1, 2, 3, 4, 5, 6}.
• Sigma algebra: Power set or smaller set based on events of interest.
• Probability measure: Assigns P = 1/6 for each outcome.

Probability Estimation Rules

• Bonferroni's Inequality: P(A ∩ B) ≥ P(A) + P(B) - 1.

• Useful when individual probabilities are hard to calculate.

• Union Bound: P(A1 ∪ A2 ∪ ...) ≤ ∑ P(Ai).

• Conditional Probability: P(A | B) = P(A ∩ B) / P(B) if P(B) > 0.

Example Problem: Coin Toss

• Tossing a coin twice; find P(both heads | at least one head).
• Elementary outcomes: {HH, HT, TH, TT}.
• Use conditional probability formulation.

Bayes' Theorem

• Rearranging conditional probabilities:
  • P(A | B) = (P(B | A) * P(A)) / P(B).
• Useful in situations where direct calculation of conditional probabilities is difficult.

Example Problem: Student Answering Questions

• Define events K (knows answer) and C (answers correctly).
• Use Bayes' theorem to find P(K | C).

Independence of Events

• A and B are independent if P(A ∩ B) = P(A) * P(B).
• Generalization for multiple events: independence holds for subsets.
• Conditional Independence: A and B are independent given C if P(A ∩ B | C) = P(A | C) * P(B | C).

Example: Admission into Programs

• Admission based on GATE scores; knowledge of admission at one institution does not inform about another under certain conditions.