Multi-Petabyte Observational Data Mesh Working Session
Hands-On Architecture Clinic on Zero-Copy Lakehouses
Designed specifically for principal data platform architects and distributed query engineers, this full-day intensive working session delivers an exhaustive technical clinic on eliminating batch ETL pipelines using Apache Iceberg, Arrow Flight SQL, and DuckDB core engines.
Participants will work with real 10-petabyte synthetic telemetry datasets to diagnose query planner bottlenecks, evaluate partition pruning heuristics, and configure peer-to-peer cache fabrics across multi-cloud object stores.
Practical Clinic Curriculum
The working session combines deep architectural lectures with guided terminal lab exercises:
- Direct NVMe & Object Storage Streaming: Implementing asynchronous io_uring pipelines to maximize storage bus bandwidth during scanning phases.
- Schema Evolution & Time-Travel Consistency: Managing high-frequency ACID snapshot commits across tens of thousands of streaming micro-writers without table lock contention.
- Cross-Cloud Query Federation: Routing queries dynamically based on egress costs, local cache warmth, and regional data residency constraints.
Prerequisites & Equipment
Attendees should possess familiarity with distributed systems concepts and comfortable navigation of SQL execution plans. High-memory remote cluster environments and pre-configured telemetry repositories will be provisioned for all registered delegates.
