Data Science: An Introduction

Introduction to Data Science

• Instructor: Barton Poulson
• Course Focus: Non-technical overview of Data Science
  • Data Science perceived as technical and intimidating
  • Emphasizes that it's creative and about gaining insights

Core Ideas of Data Science

• Uses tools from coding, statistics, and math
• Goal: Extract insights from data
• Inclusive analysis: Consider all data, even when it doesn’t fit standard methods

Demand for Data Science

• High Demand: Significant need for data scientists and data-savvy managers
  • Harvard Business Review: “Sexiest Job of the 21st Century”
  • McKinsey Global Institute projections
    • Need for 140,000 to 190,000 data scientists
    • Need for 1.5 million data-savvy managers in the US
• Job Market
  • Glassdoor: Data Scientist is the #1 job with high salary ($116,000+)
  • LinkedIn: Skills in statistical analysis and data mining highly sought globally
• Economic impact: High pay and demand make it a lucrative career choice

Defining Data Science

• Data Science Venn Diagram by Drew Conway
  • Three main areas:
    • Coding/Programming (Hacking)
    • Statistics/Mathematics
    • Domain Expertise
  • Intersections
    • Coding + Stats = Machine Learning
    • Stats + Domain Knowledge = Traditional Research
    • Coding + Domain Knowledge = “Danger Zone”

Data Science Process (Pathway)

1. Planning
   • Define goals
   • Organize resources
   • Coordinate team
   • Schedule project
2. Data Preparation
   • Data acquisition and cleaning
   • Data exploration
   • Data refinement
3. Modeling
   • Create statistical models
   • Validate and evaluate models
   • Refine models
4. Follow-up
   • Present findings
   • Deploy models
   • Revisit and archive models

Roles in Data Science

• Engineers: Focus on infrastructure
• Big Data Specialists: Handle large datasets and machine learning
• Researchers: Domain-specific research
• Analysts: Day-to-day business analytics
• Business People: Frame questions and manage projects
• Entrepreneurs: Data-driven startups
• Full Stack Unicorn: Rare individuals excelling in all areas

Teams in Data Science

• Collaboration and combining skills is key
• Example: Two people with complementary skills forming an ideal team

Contrasting Data Science with Other Fields

1. Big Data
   • Big Data vs. Data Science
   • Big Data Science: Combination of both fields
2. Coding/Programming
   • Coding is fundamental, but data science includes stats
3. Statistics vs. Data Science
   • Different backgrounds and focuses
4. Business Intelligence (BI)
   • BI uses simple analytics for practical decision-making

Ethical Issues in Data Science

• Privacy: Confidentiality of data
• Anonymity: Ensuring individuals cannot be identified
• Copyright: Legality of data usage
• Data Security: Protecting data from breaches
• Bias: Avoiding unintentional prejudice in algorithms
• Overconfidence: Recognizing limitations and avoiding blind trust in data

Methods in Data Science

• Sections:
  • Sourcing (Getting data)
  • Coding
  • Math
  • Stats
  • Machine Learning
• Goal: Insight over tech

Data Sourcing

• Methods:
  • Using existing data
  • APIs
  • Web scraping
  • Creating new data
• Quality Check: Importance of data quality and metrics

Coding in Data Science

• Key Tools:
  • R: Specific for data, widely used
  • Python: General-purpose, well-adapted for data
  • SQL: Databases
  • Other languages: C/C++, Java, Bash, Regex
• Applications: Excel, Tableau, SPSS, JASP
• Web Data: HTML, XML, JSON

Mathematics in Data Science

• Importance:
  • Determines appropriate procedures
  • Diagnosing and fixing issues
• Key Areas:
  • Elementary Algebra
  • Linear Algebra
  • Systems of Linear Equations
  • Calculus (Optimization)
  • Big O (Order of functions)
  • Probability and Bayes’ Theorem

Statistics in Data Science

• Functions:
  • Summarizing data
  • Generalizing from samples
• Exploration: Graphical and numerical exploration of data
• Inference
  • Hypothesis Testing
  • Estimation (Confidence Intervals)
• Feature Selection: Choosing informative variables
• Model Validation: Ensuring models generalize well
• Handling Common Problems: Non-normality, Non-linear relationships, Multicollinearity, Missing data