Azure Data Lake Storage Introduction

Overview

• Azure Data Lake Storage provides a set of capabilities for big data analytics, built on Azure Blob Storage.
• Converges capabilities of Azure Data Lake Storage Gen1 with Azure Blob Storage to offer file system semantics, file-level security, and scalability.
• Supports massive data management with high throughput and high availability/disaster recovery capabilities.

What is a Data Lake?

• A Data Lake is a centralized repository for storing all data types, both structured and unstructured.
• Enables storage and analysis of data in its raw format, without conforming to a specific structure.
• Azure Data Lake Storage is a cloud-based solution for storing large amounts of data in any format, facilitating big data analytical workloads.

Data Lake Storage

• Not a dedicated service but a set of capabilities within the Azure Blob Storage service.
• Enabled by activating the hierarchical namespace setting.
• Key capabilities include:
  • Hadoop-compatible access
  • Hierarchical directory structure
  • Optimized cost and performance
  • Finer grain security model
  • Massive scalability

Hadoop-compatible Access

• Designed to work with Hadoop and frameworks using the Apache Hadoop Distributed File System (HDFS).
• Uses Azure Blob File System (ABFS) driver for direct access to Azure Blob Storage data.
• Compatible with data analysis frameworks like Apache Spark and Presto SQL.

Hierarchical Directory Structure

• Organizes files and objects in a hierarchy of directories and subdirectories.
• Enables single atomic metadata operations for actions like renaming or deleting directories.

Optimized Cost and Performance

• Priced at Azure Blob Storage levels, leveraging automated lifecycle management and object tiering.
• High performance without needing data transformation before analysis.

Finer Grain Security Model

• Supports Azure role-based access control (RBAC) and POSIX access control lists (ACLs).
• Offers additional security settings specific to Azure Data Lake Storage.
• Ensures data at rest is encrypted.

Massive Scalability

• No limits on account sizes, file sizes, or data amounts in the data lake.
• Supports file sizes from kilobytes to petabytes.
• Designed for rapid scaling to meet workload demands.

Built on Azure Blob Storage

• Data persists as blobs managed by Azure Blob Storage service.
• Data Lake Storage capabilities enhance Blob Storage for big data workloads.
• Most Blob Storage features are supported, with some in preview or unsupported.

Documentation and Terminology

• Distinct sections for Data Lake Storage and Blob Storage content.
• Slight terminology differences (e.g., "blob" vs. "file").

See Also

• Training modules, best practices, known issues, and multi-protocol access related to Azure Data Lake Storage.

Additional Resources

• Training modules on Azure Data Lake Storage Gen2.
• Events like AI Skills Fest Challenge.