Notes on NPUs and AI in Technology

Introduction

• Video sponsored by Skillshare.
• Discussion of on-device AI: common in recent tech announcements.
• Terms: Apple Intelligence, Microsoft Recall, Meta AI, etc.
• Focus on NPUs (Neural Processing Units): Are they effective or just hype?

Overview of NPUs

• NPUs are components in modern devices for running AI.
• Example: Apple's M3 Max chip as an SoC (System on a Chip).
  • Components include CPU, GPU, display engines, I.O., and NPU (Neural Engine).
• Expectations vs. Reality: NPUs are smaller than other components despite the hype.

Importance of NPUs

• Size correlates with importance: larger in smaller devices (e.g., smartphones) and less significant in larger devices (e.g., PCs).
• NPUs have been common in smartphones for 7 years; PCs are just starting to adopt them.

Understanding Accelerators

• Concept of Accelerators: Specialized hardware to perform specific calculations more efficiently than general-purpose CPUs.
• Examples:
  • GPU: Handles graphics calculations efficiently by working in parallel.
  • Other Accelerators: Video encoders, image signal processors, etc.

What is an NPU?

• NPUs are dedicated to running the math behind neural networks.
• Applications:
  • Auto-complete, face detection, voice dictation, sensor data processing.
  • Tim Cook claims 200 neural models run on iPhones.

Basic Math Behind Neural Networks

• Simple example: "Martin's Image Recognition Machine (MIM)"
  • 4 pixel inputs; output indicates if there's a diagonal line.
  • Neurons: Memory cells arranged in layers, connected with weights.
• Process: Multiply input values by weights, add to neurons, generate output (1 for presence of line, 0 for absence).

Real Neural Networks

• Complex networks can have billions of neurons and parameters.
• Training happens on servers; optimization is key for NPU functionality.
• Required capabilities for NPUs:
  1. Parallel calculations with many simple cores.
  2. Large RAM to load models.
  3. Fast cache for storing results.
  4. Accept lower precision to speed up calculations.

Comparison with GPUs

• GPUs: Already primary AI chips for peak performance with dedicated AI hardware (e.g., NVIDIA's Tensor Cores).
• NPUs: Focus on power efficiency rather than peak performance.
• Examples of efficient tasks for NPUs:
  • Continuous background tasks like crash detection and heart rate monitoring.

Potential Uses and Limitations of NPUs

• Examples of proposed uses for NPUs:
  • Real-time audio captioning, translation, and video effects (background blurring, echo cancellation).
  • Recall: a Windows system that uses image recognition for creating a searchable database efficiently.
• Uncertainty about consumer acceptance of intrusive systems powered by NPUs.

Conclusion

• Growing interest in creativity amidst generative AI advances.
• Learning resources available through Skillshare for various creative skills.