Beginners Guide to Contrastive Learning

What is Contrastive Learning?

• Contrastive Learning: A machine learning paradigm using unlabeled data points to teach models which points are similar/different.
• Samples contrasted: Points from same distribution are brought closer; different distribution points are separated.

Importance of Contrastive Learning

• Supervised Learning: Relies on large labeled datasets; high-quality labels are crucial.
• Challenges: Labeled data is expensive/time-consuming to obtain (e.g., biomedical imaging).
• Need for less supervision: Techniques like Semi-Supervised, Unsupervised, and Self-Supervised Learning reduce reliance on labeled data.

How Contrastive Learning Works in Vision AI

• Mimics human learning: Infers similarities/differences without explicit labels.
• Framework: Uses anchor, positive, and negative samples to learn similarities/differences in data.

Methods in Contrastive Learning

Instance Discrimination Method

• Transforms whole images for positive samples; any other dataset image serves as negative sample.
• Uses augmentations like color jittering, rotation, flipping, noising, affine transformations.

Image Subsampling/Patching Method

• Breaks images into patches; uses patches as samples for learning.

Contrastive Learning Objectives

Loss Functions

1. Max Margin Contrastive Loss: Maximizes separation between different-class samples, minimizes between same-class samples.
2. Triplet Loss: Involves anchor, positive, and negative samples simultaneously.
3. N-pair Loss: Extends triplet loss with multiple negative samples.
4. InfoNCE: Uses noise contrastive estimation.
5. Logistic Loss: Simple convex loss function.
6. NT-Xent Loss: Involves temperature-scaled cross-entropy.

Supervised vs. Self-Supervised Contrastive Learning

• Supervised: Uses labels to generate positive samples; helps align similar samples in latent space.
• Self-Supervised: Relies on data augmentation for learning; may misplace similar samples in latent space.

Contrastive Learning Frameworks

1. SimCLR: Maximizes agreement between augmented versions using contrastive loss.
2. NNCLR: Uses nearest neighbors as positive samples.
3. ORE: Detects and learns unknown objects incrementally.
4. CURL: Jointly learns contrastive representations with reinforcement learning.
5. PCRL: Self-supervised learning for medical imaging.
6. SwAV: Uses cluster assignments between views.
7. MoCo: Uses a dynamic dictionary method.
8. Supervised Contrastive Segmentation: Enforces similarity in pixel embeddings.
9. PCL: Bridges contrastive learning with clustering.
10. SSCL: Addresses aspect detection in NLP using contrastive learning.

Applications of Contrastive Learning

Semi-supervised Learning

• Utilizes both labeled and unlabeled data; improves label efficiency.

Supervised Learning

• Uses class labels effectively to enhance contrastive learning.

Natural Language Processing

• Challenges in text augmentation; uses methods like back-translation and lexical edits.

Computer Vision

• Applied in video sequence prediction, object detection, semantic segmentation, remote sensing, audio similarity.

Summary

• Contrastive Learning excels in self-supervised learning, enhancing existing supervised methods.
• Focuses on juxtaposing data samples to refine embeddings based on class similarities/differences.
• Continues to evolve towards minimal supervision methods for better performance than traditional supervised learning.