Why the Frontier Ecosystem must be Open — Matei Zaharia and Reynold Xin, Databricks

We’re excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers!

Everyone is still talking about Satya’s Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today.

From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work.

We go deep on Omnigent: Databricks’ open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness.

Then Reynold walks through Databricks’ database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks’ infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top.

Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn’t need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over.

Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it’s the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it’s doing work?”

If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly.

Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents.

Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve!

We discuss:

* Why Databricks built Omnigent as a meta-harness above existing AI agents

* Why coding agents and custom enterprise agents need the same infrastructure

* The common API for agent sessions, files, streams, tool calls, and cancellation

* Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter

* Why Databricks open-sourced Omnigent instead of keeping it proprietary

* Databricks’ internal agent usage, cloud sandboxes, and coding workflows

* The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast

* Why agent security needs contextual and stateful policies

* How an agent could read confidential docs, install a compromised npm package, and leak data

* Why spend control matters when an agent can burn $500 reading logs

* Startup opportunities around coding-agent analytics, quality, skills, and spend

* LTAP, Lakebase, and why Databricks wants to rethink the database stack

* OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m.

* Why HTAP has historically been the holy grail of database engineering

* Why Databricks thinks LTAP is “HTAP done right”

* How writing transactional data into column-oriented formats changes analytics

* Why agents need live operational context from databases, not just telemetry

* How Databricks prototypes strategic systems without endless process

* Enterprise vs tech customers, governance, procurement, and DIY culture

* The “second system syndrome” risk of rewriting a database engine

* Building a database engine from a decade of traces and quadrillions of data points

* Why vector databases should never have been a separate category

* Why open formats and AI changed the race with Snowflake

* The Mosaic story, DBRX, Genie, document parsing models, and specialized model training

* Why model customization and RL fine-tuning may become mainstream

* Why “get the data there, slap some agent on top” may rewrite traditional software

Matei Zaharia

* LinkedIn: https://www.linkedin.com/in/mateizaharia

* X: https://x.com/matei_zaharia

Reynold Xin

* LinkedIn: https://www.linkedin.com/in/rxin

* X: https://x.com/rxin

Databricks

* Website: https://www.databricks.com

* X: https://x.com/databricks

Timestamps

00:00:00 Introduction

00:02:22 Omnigent and the Agent Infrastructure Layer

00:08:39 Agent Clouds, Common APIs, and Open Source

00:16:52 Databricks Scale and Internal AI Workflows

00:18:03 Agent Security, Governance, and Spend Controls

00:27:34 LTAP and the Database Dream

00:30:30 CDC, HTAP, and Why Data Pipelines Break

00:34:05 Lakebase, Parquet, and Live Data for Agents

00:36:47 Databricks’ Culture of Fast Prototyping

00:43:40 The Dream Engine and Rewriting the Database Stack

00:51:02 Vector Databases, Query Engines, and LTAP

00:52:36 Databricks vs Snowflake

00:57:48 Mosaic, DBRX, Genie, and Specialized Models

01:03:11 Context, AI Runtime, and RL Fine-Tuning

01:06:15 Why Data + Agent