Lecture: Challenges and Real-World Applications of Machine Learning and Large Language Models

Introduction

• Presenters: Nico (Research Scientist, Founder of CometML) and Dr. Doug Blank (Generative Media Expert, Founder of CometML)
• Background on Comet: MLOps software to support machine learning (ML) model building and deployment
• Focus: Real-world applications, challenges, and case studies related to ML and Large Language Models (LLMs)

Case Studies in Industry

Case Study 1: User Verification Model on a Dating App

• Problem: Verify user-uploaded photos to detect non-genuine profiles
• Dataset: Manually labeled images from the app
• Model: Neural network classifier
• Issue: Degradation in model performance over time (10%-15% decrease)
  • Initial hypothesis: Data drift
  • Retraining the model did not resolve the issue
  • Actual problem: New iPhone camera with higher resolution images
  • Solution: Added additional layers to the neural network to handle higher resolution images

Lessons Learned

• Regular system reviews and monitoring are crucial
• External factors (e.g., new devices) can significantly impact model performance

Case Study 2: E-commerce Recommendation System

• Problem: List and ad recommendations to maximize the likelihood of a click and subsequent purchase
• Dataset: Embeddings of search queries, item favorites, views, add-to-carts, and purchases
• Model: Embedding model for retrieval and ranking
• Issue: Difficulty in surpassing the performance of the production model
  • Hypothesis: Seasonal trends, different local optimal points in training, consumer segmentation, and data distribution shifts
  • Actual problem: New model trained on embeddings produced by the production model
  • Solution: Built multiple retrieval agents in production to diversify embedding data for new training

Working with Large Language Models (LLMs)

Introduction to LLMs

• Transformative models like GPT (Generative Pre-trained Transformer)
• Core idea: Encoding large blocks of text and iterative generation of outputs
• Issues: Hallucinations, inaccuracies, and unrealistically varied outputs

Common LLM Failures and Diagnostics

• Example 1: Decoding incorrect results (ROT13 cipher decoding issue)
• Example 2: Inconsistent feedback in essay scoring
• Example 3: Hallucinated information (e.g., fictional identities)

Key Insights

• LLM outputs are highly dependent on prompt engineering and probabilistic selection
• Sensitive to low-frequency data and context
• Potential solutions involve adjusting model parameters and monitoring probabilities of output

Ethical Considerations and Practical Use Cases

• Ethical implications: Misinformation, biased outputs, and negative social impacts
• Practical applications: Coding assistance, content summarization, and creative inspiration
• Issues in implementing LLMs for critical decisions or external customer interactions

Effective Production Strategies

• Modular framework for tracking experiments and metrics
• Use alerts, logging, and continuous monitoring
• Regular updates and maintenance to cope with changing real-world applications

Q&A Highlights

• Grade reversal hypothesis: Observing impact of grading sequence on textual output
• Methods to reduce hallucinations in LLMs via user feedback and fine-tuning
• Challenges in monitoring model consistency and performance
• Strategies in adapting ML models to external changes, like improved camera technology

Conclusion

• Importance of MLOps and continuous model monitoring
• Unpredictable nature of real-world applications prompts the need for regular review
• Ethical deployment and adaptability are key to successful ML and LLM application

Final Words from Presenters

• Importance of using comprehensive tools for ML model management (mention of CometML)
• Encouragement to explore various tools and establish best practices

[Applause]