Lecture Notes: Big Data and Its Challenges

Introduction to Oracle Consultation

• Historical Context: In ancient Greece, oracles were consulted for major life decisions (e.g., marriage, voyages, military actions).
  • Process involved a trance state, leading to predictions.

Modern-Day Oracle: Big Data

• Current Use: Big data, machine learning, and AI are the modern equivalents of oracles.
  • Used for logistics, medical predictions, sales forecasting.
  • Example: Weather predictions via apps like Dark Sky.
• Industry Insight: The big data industry is valued at $122 billion but has low returns.
  • Over 73% of big data projects aren't profitable.
  • Companies face challenges in leveraging data for better decision-making.

Challenges in Using Big Data

• Observation: Many organizations fail to utilize big data effectively.
• Example: Nokia missed the smartphone trend despite qualitative insights pointing towards its rise.
  • Nokia relied on large data sets, dismissing smaller, qualitative data.

Quantification Bias

• Definition: The tendency to value measurable data over qualitative insights.
  • Leads to overlooking critical insights not captured numerically.

Case Study: Temple of Apollo

• Geological Findings: The oracle’s predictions were influenced by ethylene gas from fault lines.
  • Temple guides interpreted the oracle’s predictions, integrating qualitative insights.
• Application to Big Data: Big data systems require 'temple guides' (ethnographers, user researchers) to provide qualitative context ('thick data').

Thick Data vs. Big Data

• Thick Data: Data that includes stories, emotions, and interactions.
  • Provides depth and context.
• Integration Benefits: Forms a more complete data picture.
  • Example: Netflix utilized thick data to understand binge-watching habits and transformed user experience.

Implications of Big and Thick Data Integration

• Impact on Industries: Beyond media consumption, has life-or-death implications.
  • Issues like predictive policing and AI biases highlight the need for thick data.
• Future Directions: Encourage better use of tools to integrate big and thick data for improved decisions.

Conclusion

• Call to Action: Use better tools to integrate data sources for making informed decisions across different fields.
  • Emphasizes collaboration between quantitative and qualitative insights to avoid the pitfalls of missing crucial data points.