The Databricks Data Engineer

Jakub Lasak
  • 0,0 (0)
  • Технологии
  • Еженедельно

Short episodes on the Databricks skills that get data engineers hired and promoted. One theme per episode, built for a commute: how senior engineers make the calls, what interviews actually test, and the career moves that change your level. By Jakub, a Databricks Data Engineer (ex-Uber).

  1. -4 дн.

    Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders

    Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right. The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas. In this episode: - What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fix - Why dropping bad rows at bronze feels clean and quietly decides how far back you can rebuild - When skipping a layer is the right call, and why seven layers is not a safer bet than three - The question to ask about every gold table before your next architecture review This episode is for Databricks data engineers who build or inherit a medallion architecture and have wondered what silver is actually for. Whether you're designing a new lakehouse or maintaining one that looks textbook, you'll walk away with a way to check that each layer does a job no other layer does, and to defend it with more than "it's best practice." --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders

  2. 2 нояб.

    How the always-sure Databricks engineer caps their own promotion (and never sees it)

    Mark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved. Sarah says what she'd bet on, what she doesn't know yet, and when she'll know. Sarah claimed less. The decision goes to Sarah. Nobody tells Mark why. The instant-answer reflex that got him to mid-level is the same one that quietly keeps him there, and there's a name for what he's paying. In this episode: - Why a fast, confident answer is right early in a Databricks career, and when it stops paying - What the room needs from you when nobody knows for sure - The difference between being decisive and sounding certain - One sentence, two parts and a date, to carry into your next design review This episode is for Databricks data engineers who answer fast and still watch the risky migration and the design review invite go to someone else. You'll walk away with a way to make your "I'm sure" mean something again. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    How the always-sure Databricks engineer caps their own promotion (and never sees it)

  3. 26 окт.

    Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)

    Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them. A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy middle version, same data collides, different data is fine, falls apart the first time the error shows up. In this episode: - What Delta actually does when the second job tries to commit to a table that moved under it - Why two jobs writing completely different months can still collide, and what that error message is really telling you - Why scheduling jobs hours apart is a lock you built by hand, and the cheaper fix - When a retry quietly does damage instead of fixing things This episode is for Databricks data engineers who own a Delta table with more than one writer: a nightly merge, a backfill, a plain append. Whether you're designing a new pipeline or defending one in a design review, you'll walk away with a way to say which endings each writer can produce, instead of hoping. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)

  4. 19 окт.

    Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted

    The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real? The catalog has something to show for every field: an owner, a check mark, a health indicator that says healthy, a beautiful lineage graph. None of them answers that question. Unity Catalog might be the best thing your platform team ever shipped, and it still can't make one person believe a number. The missing piece isn't a feature. It's the habit of treating every trust complaint like one. In this episode: - Why an owner, a certified check mark and a healthy indicator can all look right while the number is still wrong - Why the most-read table in your catalog might be the least trusted one - What a metric view settles when two teams define revenue differently, and what it leaves to humans - The one question to ask before you touch the platform for any governance request This episode is for Databricks data engineers who work in Unity Catalog and keep getting governance requests another tag won't fix. Whether you just finished the migration or you're the one answering "which number is real?" in Slack, you'll walk away with a way to sort every request before anyone opens a ticket. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted

  5. 12 окт.

    Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through

    Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one. It's tempting to call that ignorance or politics. It's neither. The better proposal was arguing merit to someone who gets asked a completely different question upstairs, in a meeting the engineer who wrote it will never be in. In this episode: - The fear that filters every proposal on your director's desk, and why it isn't fear of change - The question a tech lead, an engineering manager and a director each get asked, and why pitching a director like a tech lead loses - Three kinds of data leader, and how to tell which one you work for from the first question they ask about any proposal - Three things to add to any proposal with a real spread, so it gets funded without getting duller - A two-sentence test to run before anything goes upstairs This episode is for Databricks data engineers who have watched a better proposal lose to a duller one and couldn't explain why. Whether you're the mid-level engineer writing the pitch or the senior trying to get a platform change funded, you'll walk away able to read your leader's decisions and write proposals they can say yes to. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through

  6. 5 окт.

    Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters

    Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill. The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow. In this episode: - Why more workers and more shuffle partitions both leave the stuck task exactly where it was - Where heavy keys come from, and the one source that is usually a bug in a performance costume - Why plenty of skewed aggregations never slow down, and where adaptive query execution on Databricks does and doesn't split the load - When salting pays, when a different approach beats it, and when the right call is to leave skew alone This episode is for Databricks data engineers whose reflex for a slow Spark stage is a bigger cluster. Whether you're mid-level and tired of paying for compute that changes nothing, or senior and about to be asked "how do you handle data skew?" in an interview, you'll walk away able to tell a skew problem from a capacity problem before you touch the cluster settings. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters

  7. 28 сент.

    Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review?

    Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish. The packet is full of work and empty of decisions. Sam thinks what he's missing is a chair at the architecture review table. It isn't. In this episode: - What the architecture review was ever supposed to prove, and why the room was never the proof - The one test to run on any call you've made, with no meeting, no invite and no template required - The two paragraphs that turn a design doc from a build plan into a decision somebody can audit nine months later - How to run the whole thing in a Slack thread on a four-person team with no review process For Databricks data engineers building a case for senior, especially where architecture reviews are rare or don't exist. You'll walk away able to tell work you did apart from a decision you can prove you owned. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review?

  8. 21 сент.

    Why Databricks serverless exists, and which of your jobs should stay on classic clusters

    Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story. Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option. In this episode: - Why the cluster wait was only the receipt, and what classic Databricks compute never actually solved - Why Databricks can keep compute warm when your own team never could, and what that shows up as in the rate - What "serverless" means here, jobs and notebooks, and why a serverless SQL warehouse is a different conversation - The thing a fixed cluster was quietly doing for your budget that serverless stops doing, and why the policy you're thinking of doesn't cover it - Three tests to run on your own workloads this week, plus the ones that can't move at all no matter what the math says This episode is for Databricks data engineers who have serverless sitting in a dropdown and no real argument either way. Whether you're defending a migration in an architecture review or explaining next month's bill, you'll walk away able to say which of your jobs should move, which should stay on classic, and why. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Why Databricks serverless exists, and which of your jobs should stay on classic clusters
См. все (33)

Об этом подкасте

Short episodes on the Databricks skills that get data engineers hired and promoted. One theme per episode, built for a commute: how senior engineers make the calls, what interviews actually test, and the career moves that change your level. By Jakub, a Databricks Data Engineer (ex-Uber).

Информация

  • Автор
    Jakub Lasak
  • Годы выхода
    2026
  • Выпуски
    33
  • Ограничения
    Без ненормативной лексики
  • Авторские права
    © Jakub Lasak
  • Сайт подкаста
    The Databricks Data Engineer

Вам может также понравиться