Nike had 450 different reds. University Red, Jordan Red, and 448 more. Query for "red shoes" and you either hit a graph database of colors first or you scan every string value in the column. Scott Haines spent years inside that problem, and it is why he cares about catalogs, tags, and semantics more than most people who talk about them. Part 2 picks up where Part 1 left off. Scott is in developer relations at Databricks, working on Delta Lake, Iceberg, and Unity Catalog, and nine months into Rust. He is the author of Modern Data Engineering with Apache Spark and a co-author of Delta Lake: The Definitive Guide. We talk about what a catalog actually is (where data lives and who owns it, which matters more once everybody's agents are querying too), where semantics go when "revenue" means three things in one company, metric views landing in Spark 4.2 and Unity Catalog 0.6, and which layers of the stack have to stay open: catalogs, table formats, and increasingly governance itself. Then the part Scott is most blunt about: the notebook that ends in DROP TABLE, the HDFS cluster at Twilio with no undo, dry-run-by-default as a design rule, the 60-page governance doc nobody read at Nike, and why he reached for the Cedar policy language there and is working with it again now. It ends on the open question: when an agent is a proxy for a lot of processes, whose permissions does it inherit? Chapters 00:00 What Scott is working on now: Spark, Iceberg, Delta, DataFusion, Rust 00:54 Nine months into Rust, and hiring "a good engineer" 02:21 Software primitives are finally commodities 03:34 What a catalog is: tribal knowledge before, discovery and ownership after 05:08 Everybody's agents: certified data and the source of truth 06:23 From infrastructure abstractions to semantics 07:00 450 reds at Nike: ontologies, a graph of colors, tags vs scanning columns 09:00 Metric views in Spark 4.2, and SQL snippets passed around in Slack 10:37 Unity Catalog 0.6 metric views and governance dictionaries 12:56 Trust but verify, minus the verify 13:10 What has to be open: catalogs, Iceberg REST, table formats, compute 14:32 Apache Gravitino and the lock-in problem 14:44 Open governance: exchanging policies, tags, and column masks 16:16 Why governance: the notebook that drops a table in prod 17:04 HDFS with no undo, and dry run as the default 19:00 Service principals, Vault, and config-based production at Nike 20:00 Sixty pages nobody reads, and the Cedar policy language 21:00 Agents as proxies: whose permissions do they inherit? Part 1: https://youtu.be/4nWeC_NEYmw Scott on LinkedIn: https://www.linkedin.com/in/scotthaines/ Delta Lake: The Definitive Guide: https://www.oreilly.com/library/view/delta-lake-the/9781098151935/ Modern Data Engineering with Apache Spark: https://link.springer.com/book/10.1007/978-1-4842-7452-1 Data for AI is produced by Datastrato, the original creator of Apache Gravitino. Sponsored by Neo4j. Join the community: https://datastrato.ai/discord Next meetup: https://lu.ma/dataforai Watch on YouTube: https://www.youtube.com/@data-for-ai
