Amazon Web Services

In this AWS re:Invent 2023 session, Netflix engineers Ashwin Kayyoor and Rakesh Veeramacheneni share their journey of modernizing Netflix's massive data lake using Apache Iceberg. They discuss the challenges of managing an exabyte-scale data warehouse and the transition from a Hive-based system to an Iceberg-only architecture. The presentation covers the development of custom tooling, ecosystem services, and unique features like secure Iceberg tables and the Iceberg REST catalog. The speakers detail the migration process, including strategies to minimize data movement and user friction while ensuring business continuity. They also highlight the benefits of Iceberg, such as ACID transactions, rich metadata layers, and improved query performance. The talk provides valuable insights for organizations looking to scale their data infrastructure and leverage open-source table formats for efficient data management.

customer-stories
product-information
skills-and-how-to
media-and-entertainment
data
Show 7 more

Up Next

VideoThumbnail
1:03:17

Transforming Business with AWS Analytics: Building Robust Data Foundations for AI and Innovation

Nov 22, 2024
VideoThumbnail
59:17

How Netflix Leverages AWS for Multi-Region Cache Replication: Insights from EVCache Architecture

Nov 22, 2024
VideoThumbnail
6:58

Minimizing OpenSearch Storage Costs: Leveraging Amazon Q for Architectural Optimization and Code Refactoring

Nov 22, 2024
VideoThumbnail
58:49

AWS Clean Rooms ML and Differential Privacy: Revolutionizing Secure Data Collaboration

Nov 22, 2024
VideoThumbnail
4:02

Streamlined Next.js 13 Deployment: AWS Amplify Hosting's Zero-Config Solution for Faster Builds and Seamless CloudWatch Integration

Nov 22, 2024