This Week in Open Lakehouse

Lisa Cao and Scott Haines
  • 5.0 (1)
  • Technology

This Week in Open Lakehouse is a weekly data engineering podcast discussing the latest in open source news, including query engines, data catalogs, open table formats, orchestrators, and streaming technologies. Produced by the Databricks DevRel team. https://openlakehouse.io/

Episodes

  1. 4d ago

    DuckDB 2.0, External Lookup Joins, Arrow Database Connectivity, and OpenCode 2.0

    This week, Scott and Lisa recap a busy week across the open lakehouse ecosystem, from DuckDB 2.0 and Iceberg’s evolving V4 spec to new proposals for Spark, Flink, and Airflow. They also discuss catalog security and project governance, plus updates to Apache Comet, OpenCode, and Omnigent. Together, they highlight the latest developments, security fixes, and community changes shaping the open-source data landscape. DuckDB 2.0 and DuckLake 1.1, including new lakehouse features, the Quack client-server protocol, asynchronous I/O, and breaking changes to plan for when upgrading.Unity Catalog and Lakekeeper changes, including stricter privilege checks, distributed cache invalidation, and updates to credential handling.Paimon’s new Terraform/OpenTofu provider for managing tables, roles, and policies as code.Iceberg V4 discussions around file types, multimodal data, efficient column updates, single-file commits, and metadata overhead from small writes.An Arrow security fix for a vulnerability involving KMS URLs embedded in encrypted Parquet files, with a recommendation to upgrade to Arrow 26.0.0.Apache DataFusion Comet’s top-level project proposal, including its growing contributor community and a proposed limit on concurrent open pull requests.StreamFusion’s possible move toward becoming a Flink subproject and what shifting project ownership can mean for users and contributors.Spark proposals for ADBC connectivity and external lookup joins, including early prototype benchmarks showing 3–10x read throughput over JDBC on column-oriented databases.Airflow’s inbound webhook proposal and how webhooks could support event-triggered workflows.OpenCode 2.0’s background agents, execution modes, and plugin support, along with Omnigent 0.19.0 and 0.19.1 updates.New Paimon, Flink, and Arrow committers, along with a new Iceberg PMC member.

  2. Oct 23

    Unity Catalog 0.7, Paimon 2.1, Hudi 1.1.2, Arrow 26.0, and new ASF top-level projects

    Scott Haines and Lisa Cao are back after a flu-delayed skip, just in time for Week 42 — which, yes, is our favorite week - "Hitchhiker’s Guide" week. We tear through a stacked open lakehouse release week, led by Unity Catalog 0.7.0, shipping native Iceberg REST catalog support, with Paimon, Hudi, Arrow, MLflow, and OmniGent close behind. In this episode Cold open / Week 42. Scott had the flu, last week vanished into the void, and the “answer to life, the universe, and the open lakehouse” ...Unity Catalog 0.7.0. Native Iceberg REST catalog support, plus a playground so you can try classic Unity Catalog and the new Iceberg path without assembling a lab from spare parts.Production hardening. Live, ready, and metrics endpoints; multi-instance setups behind a load balancer; Spark support through 4.1, with 4.2 still waiting on upstream Iceberg.Apache Paimon 2.1.0. Multimodal and table-format upgrades: scan planning, Arrow reads/writes, blob collections, variant updates, indexes, and full-text search.Iceberg compatibility, no shortcuts. Paimon stays aligned with Iceberg’s published type model — including the fussy bits, like nanosecond timestamps and variant.Apache Hudi 1.1.2. Stronger Apache Flink streaming merge-on-read, 19 backported fixes, and a Flink upsert bug fix that keeps writes from quietly going missing.Apache Kafka 4.2.2. A short stop: mostly a performance-oriented release.Apache Arrow 26.0.0. The vote passed, the release is official, and Jeffrey Vo joins the PMC.MLflow 3.17.1 and 3.17.2. Decision API support, custom judges, evaluation lineage, and UI fixes for Claude Code OpenTelemetry traces.OmniGent 0.18. Bring-your-own-agents, web reading that turns pages into clean markdown, plus GitLab MR and Azure DevOps PR integrations.Community, the Apache Way, and how not to land a 4,000-line first PR The back half leaves the changelog and talks about project health: incubator graduation, contributor onboarding, CI cost, release management, and why credibility in a community still beats dropping a giant patch on day one. Lisa also flags three new ASF top-level projects: Apache Auron — a native, Rust-powered accelerator for big data query enginesApache Reqsign — a lightweight signer for cloud storage HTTP requests, without hauling around vendor SDKsApache ResilientDB — a modular, permissioned blockchain framework built on Byzantine fault-tolerant consensus

  3. Oct 13

    DataFusion Ballista 55.0, Unity Catalog Native Iceberg Support, Parquet VECTOR type, and MLflow 3.17

    In this episode, Lisa and Scott cover a dense week of open-source data infrastructure updates from Community Over Code and Data and AI Weekend. They explore new release activity across the lakehouse stack, the growing role of AI agents and policy enforcement, and the emerging standards that will shape multimodal and AI workloads. The discussion highlights the differences between hybrid and native table architectures, standardization of decimal precision and vectors, and the critical need for traceability and guardrails in agent systems. Unity Catalog 0.7, native Iceberg support, Kubernetes-style endpoints, metrics, and batch loading for large schemasDelta Lake 4.4.1, catalog-managed maintenance operations, metadata cleanup, column mapping removal, and automatic log cleanupTable-format architectures, native versus hybrid support, and Delta and Iceberg workload managementApache Arrow 26.0.0, Python compatibility updates, Parquet reader flexibility, DLPack 1.3, and the Flight SQL ODBC driverApache Parquet, vector logical types, embedding-native storage, deck float, and decimal precision standardsMLflow 3.17.0, judged feedback, fine-grained permissions, rollups, evaluation lineage, and run restore supportOmnigent 0.17.0, OIDC sign-in, policy engine updates, session-aware controls, and agent sandboxingData Fusion Comet 1.10, native Iceberg writes, Spark 4 support, and native shuffleBallista improvements, memory management, AQE, higher broadcast thresholds, and handling large I/O-heavy jobsAgent governance, contextual policy enforcement, traceability, evaluation lineage, and behavioral guardrailsIceberg REST catalog, user agent support, and identity managementApache Hudi, vector search support, and semantic storage choices

  4. Oct 6

    Iceberg 1.12, OIDC Credentials, Spark Connect Gateway, Arrow JSON Schemas, and Release Management

    In this episode, Lisa and Scott explore the latest developments across the open data ecosystem, including Delta Rust, Apache Iceberg, Apache Spark, Rust-native infrastructure, and the challenges of keeping rapidly evolving technologies interoperable. They discuss release velocity, governance, security, schema evolution, standardization, and the maintenance work required to make open-source innovation reliable in production. Delta Rust 1.0, DataFusion integration, column mapping, schema evolution, and reduced JVM dependenciesApache Iceberg 1.12, variant types, variant shredding, V4 foundations, and performance improvementsSpark security proposals, row-level filters, column masking, trusted execution engines, and policy enforcementOIDC credential propagation, short-lived credentials, identity management, and modern authenticationSpark Connect Gateway, proxy-driven architecture, multi-tenancy, routing, rate limits, and observabilityRust adoption, modular infrastructure, smaller binaries, faster startup times, and composable systemsInteroperability across Spark, Flink, DuckDB, Polars, Arrow, Iceberg, and DeltaSchema evolution, column mapping, positional assumptions, nested data, and logical versus physical representationsStandardized schemas, shared expression models, JSON representations, and reducing translation layersRelease candidates, backports, compatibility matrices, security patches, and the role of open-source maintainers

  5. Sep 29

    XML Functions, Polars 2.0, delta-rs, and Agent Thrashing Policies

    In this episode, Lisa and Scott explore recent updates in ecosystem technologies including Apache Iceberg, Apache Flink, Polars, delta-rs, highlighting their frustrations in agent workflows and a potential oversight on the importance of XML. Iceberg 1.12: REST catalogs, C++ support, equality deletes, and interoperabilityPolaris catalog governance, lineage, pagination, and deletion policiesFlink updates, including State Fun deprecation and native XML functionsPolars 2.0, streaming execution, lazy frames, and memory efficiencyDelta Rust releases, partition handling, correctness fixes, and migrationParquet performance, FastLanes benchmarks, and Arrow’s Intel macOS supportAI agent orchestration with Omnigent, Kubernetes sandboxes, Spark pipelines, and thrash detection

  6. Sep 18

    FileTypes, Language-Agnostic UDFs, Parquet Versioning, and Iceberg Updates

    In this episode, Lisa and Scott explore recent updates in data ecosystem technologies including Apache Iceberg, Apache DataFusion, Apache Parquet, and the Lakekeeper Iceberg Rest Catalog, highlighting their implications for data management and interoperability. Iceberg Data Fusion integrationParquet 2.14 release and versioningNanosecond timestamp support in ParquetIceberg 1.12 release and V4 specificationIceberg rest catalog support in Unity CatalogLakekeeper updates and multimodal support

  7. Sep 18

    Nanosecond Timestamp Precision, Iceberg-Datafusion,Spark Connect Rust Client, and Parquet Types

    The biggest throughline this week is format-layer consolidation. Apache Parquet 2.14.0 shipped as a final release, carrying chronological ordering for INT96 timestamps and Variant documentation fixes, while a parallel dev-list debate over versioning semantics and reader behavior for unsupported versions shows the community is actively thinking about long-term compatibility contracts, not just feature additions. One layer up, Lance v12 is in a sustained beta sprint, shipping six beta releases in a single week. The work spans file format stabilization (resolving the stable format to 2.2), distributed vector search in Java, in-memory WAL improvements, and IVF index optimization. This is not routine maintenance; it reads as a deliberate push to lock down a production-ready v12 surface before a GA cut. At the query engine and integration layer, two parallel votes to move iceberg-rust's DataFusion integration into the Apache DataFusion project are the most structurally interesting governance event of the week. If both votes pass, Iceberg catalog access becomes a first-class DataFusion concern rather than a satellite project, which changes how engine builders think about Iceberg adoption. Separately, the Spark Connect Rust client reached RC2 for its 4.2.0 release, and Spark 4.3.0 RC1 is in vote, keeping the Spark release cadence unusually active.

  • 7 Episodes

About

This Week in Open Lakehouse is a weekly data engineering podcast discussing the latest in open source news, including query engines, data catalogs, open table formats, orchestrators, and streaming technologies. Produced by the Databricks DevRel team. https://openlakehouse.io/

Information

  • Creator
    Lisa Cao and Scott Haines
  • Years Active
    2026
  • Episodes
    7
  • Rating
    Explicit
  • Copyright
    © 2026 Lisa Cao and Scott Haines