TF-IDF Overview

Definition

• TF-IDF: Product of Term Frequency (TF) and Inverse Document Frequency (IDF).
• Specific to a single document (d).
• IDF depends on the entire corpus.
• Commonly used in text representation.

Preprocessing Steps

• Stemming: Reducing words to their root form.
• Stop Words Elimination: Removing common words that may not be useful for analysis.
• Term Counts: Count of terms in a document for TF.
• Document Counts: Count of documents across the corpus for IDF.

Feature Vector Creation

• Each document becomes a feature vector.
• The corpus is a set of these vectors.
• Useful for data mining algorithms in classification, clustering, or retrieval.

Simple Example

Given Sentences (Corpus D)

• Document 1 (D1): "A quick brown fox jumps over the lazy dog. What a fox!"
• Document 2 (D2): "A quick brown fox jumps over the lazy fox. What a fox!"

Objective

• Analyze relevance of the word "fox" in documents D1 and D2.

Calculation Steps

1. Calculate Term Frequency (TF)

• TF Formula: ( TF = \frac{\text{Number of times term appears in document}}{\text{Total number of terms in document}} )

D1 Calculation

• Total words = 12
• Occurrences of "fox" = 2
• ( TF_{D1} = \frac{2}{12} = 0.17 )_

D2 Calculation

• Total words = 12
• Occurrences of "fox" = 3
• ( TF_{D2} = \frac{3}{12} = 0.25 )_

2. Calculate Inverse Document Frequency (IDF)

• IDF Formula: ( IDF = log\left(\frac{N}{n}\right) )
  • Where N = total documents, n = documents containing term.

Calculation for "fox"

• Documents containing "fox" = 2 (D1 and D2)
• Total documents = 2
• ( IDF = log\left(\frac{2}{2}\right) = 0 )

3. Calculate TF-IDF

• TF-IDF Formula: ( TF-IDF = TF \times IDF )

D1 Calculation

• ( TF-IDF_{D1} = 0.17 \times 0 = 0 )_

D2 Calculation

• ( TF-IDF_{D2} = 0.25 \times 0 = 0 )_

Conclusion

• Both documents have a TF-IDF of 0 for the term "fox".
• This implies equal relevance of "fox" in both D1 and D2.

Summary of Key Concepts

• TF: Raw count of a term in a document.
• IDF: Measure of how much information the term provides across the document set.
• Use in NLP: Scoring words in machine learning algorithms, automated text analysis, and information retrieval.

Final Thoughts

• TF-IDF is crucial for understanding document relevance in text analysis.
• Encouragement to keep learning and exploring more on this topic.