The Databricks Data Engineer

Jakub Lasak

Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

  1. 5日前

    How to get credit for Databricks platform work without becoming the on-call martyr

    Your pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the backbone of the team. You've also been at the same level for three years, and nobody can quite explain why. It's not bad luck, and it's not a skill gap. The exact thing everyone praises you for, being the one who can always fix it, is the reason they can't afford to move you up. In this episode: - Why being the indispensable on-call hero quietly caps the level you can reach - How one engineer systematized a whole class of Databricks incidents out of existence and got promoted for it - The difference between measuring your work in saves and measuring it in risk retired - How to make yourself removable from the hero seat without handing away your job security - The one self-test that tells you whether your recognition depends on things staying broken This episode is for Databricks data engineers who carry the pager and keep the platform standing, but keep watching less essential peers get promoted first. Whether you're the war-room legend or the quiet engineer whose prevention work goes uncredited, you'll walk away with three concrete moves to turn indispensability into a promotion case. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    How to get credit for Databricks platform work without becoming the on-call martyr

  2. 7月13日

    Why Databricks Built Delta Lake When Parquet Was Already Good Enough

    A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage. That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives one level up, in the one thing a folder of Parquet files has never had and a real table cannot live without. In this episode: - Why a folder of Parquet files was never actually a table, and what a crashed write silently does to it - The single idea that turns ACID, time travel, and concurrent writes from three separate Delta features into one - How Delta really protects you when two pipelines write at once, and why your code should expect a "no" and retry - A portable question you can point at any storage system to know in one sentence whether you can trust it - The difference between how junior and senior engineers explain why their team runs on Delta This episode is for Databricks data engineers who write to Delta tables every day and have never had to explain why it exists. Whether you're prepping for an interview or sitting in a migration meeting, you'll walk away able to explain in sixty seconds what Delta actually bought you, and why the thing that came before it was quietly lying to everyone. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Why Databricks Built Delta Lake When Parquet Was Already Good Enough

  3. 7月6日

    The Databricks pipeline that ran green for a year and was wrong the whole time

    A revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning: green. Every morning, everyone moves on with their day. For a year, nobody looked any closer. The job succeeded every single night, so everyone trusted the data. But "finished" and "correct" are two different words, and the space between them had been quietly swallowing a slice of revenue for eleven months, with no error, no warning, and not one alert. In this episode: - Why a Databricks job can run green every day and still feed wrong numbers into decisions that matter - The one question that separates "did it run" from "is it right" and cracks silent failures wide open - How an ordinary upstream schema change and an inner join can drop rows for months without tripping anything - The single correctness check that would have caught this on night one instead of month eleven - What to say in the room after an incident like this if you want leadership to trust you with the next hard problem This episode is for Databricks data engineers who own pipelines whose numbers someone actually makes decisions on. Whether you've ever said "the pipeline's fine, it's probably the dashboard" or you're staring at an all-green run history right now, you'll walk away knowing exactly what to check before you trust it again. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    The Databricks pipeline that ran green for a year and was wrong the whole time

  4. 6月29日

    How Photon actually makes your Databricks queries faster (and when it silently doesn't)

    Two engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to make a coffee and comes back to a query still running. Sam's is done before they finish reading the first Slack message. The only difference is one checkbox on the cluster called Photon. Most Databricks data engineers have that box ticked, pay a premium for it on every DBU, and could not explain in plain English what it actually does. "Photon makes it faster" isn't an answer - it's just saying the box does what the box does. And there's one thing it quietly stops doing while you keep paying for it. In this episode: - The single architectural change underneath Photon, explained without one line of code or config - A mental model that lets you explain Photon to a PM in 60 seconds flat - Why Photon flies on scans, filters, joins, and aggregations but does nothing for certain custom code - The silent fallback that charges you the fast-engine premium for slow-engine work, with nothing on screen to warn you - The one query to open in the query profile this week, and the single question to ask of it This episode is for Databricks data engineers who run Photon every day and treat it as a speed toggle they never think about. Whether you're a mid-level engineer who clicks the box and moves on, or a senior who's never checked whether the premium is paying off, you'll walk away knowing exactly how to tell when Photon is doing the work you're being billed for - and when it isn't. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    How Photon actually makes your Databricks queries faster (and when it silently doesn't)

  5. 6月22日

    The Databricks interview round nobody studies for (and almost everybody fails)

    Picture the debrief room after a Databricks loop. Two candidates went through that day. On paper, a coin flip: SQL tied, Spark internals solid, system design clean for both. Score only the rounds with a rubric and you cannot separate them. And yet the room isn't split. One gets the offer, and the thing that decided it wasn't any of the rounds they studied for. It was the conversation everyone treats as filler. There's a reason your strongest technical answers can't win it for you, and it's not the one you'd guess. In this episode: - Why the round with no whiteboard and no visible rubric is the one that decides close calls - What it actually measures, and why your best technical round can't test it - The three ways strong engineers fail it without ever noticing - Why "tell me about a decision you'd make differently" is a trap baited with your own best work - The one prep move that needs zero new facts, just a few honest walks This episode is for Databricks data engineers cleared past the technical bar who keep losing the close calls. Whether you're prepping for your next loop or wondering why a strong interview still ended in a no, you'll walk away with a specific way to rehearse the conversation everyone treats as a throwaway. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    The Databricks interview round nobody studies for (and almost everybody fails)

  6. 6月15日

    The Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)

    Your Spark job has been running for forty minutes. The dashboard shows your cluster isn't even busy. So you do the obvious thing: add more workers. And it changes nothing. Here's why. During a shuffle, Spark is barely computing at all. It's tagging every row by destination, piling rows together, spilling the overflow to disk, and hauling data across the network between executors. It's an airport rerouting every passenger's bag to a new carousel, and more baggage handlers can't speed up a single overloaded belt. In this episode: - Why your slowest wide transformation spends most of its time on logistics, not computing - The four-step model that lets you explain the shuffle to a teammate in sixty seconds - Why adding workers can make a skewed job slower, not faster - The two numbers in the Spark UI that tell you whether it's skew, partition count, or spill - The one diagnostic to run before you ever resize the cluster again This episode is for Databricks data engineers whose joins and aggregations crawl for reasons the cluster size never seems to fix. Whether you're mid-level and tired of guessing, or senior and tired of paying for compute that doesn't help, you'll walk away able to read a slow shuffle instead of throwing hardware at it. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    The Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)

  7. 6月8日

    Your Databricks data quality framework is a Yeti: everyone talks about it, nobody has seen it work

    An architecture review. A platform team is presenting their data quality setup, and honestly, it's impressive. Expectations on every ingestion table. Drift metrics on the dashboard. A dedicated alerts channel. Then a finance engineer asks the only question that counts: when did this last catch something before one of us did? Silence. That silence is the whole problem. The decks, the suites, the dashboards are everywhere. The proof that any of it actually works is somewhere else entirely. In this episode: - Why the discipline every Databricks team talks about is the one with the fewest confirmed wins - A 30-second test that tells you if your data quality framework is alive or just well documented - Why checks chosen by what's easy to write miss the incidents that actually break you - The one deadline-night decision that quietly kills more frameworks than any outage - The reframe that separates teams who get caught off guard from teams who don't This episode is for Databricks data engineers who own pipelines feeding the numbers their company argues about. Whether you've got expectations in warn mode you never reverted, or a monitoring dashboard nobody reads, you'll walk away with a way to tell whether your framework is real and what to do if it isn't. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #DataQuality #ApacheSpark #DeltaLake

    Your Databricks data quality framework is a Yeti: everyone talks about it, nobody has seen it work

  8. 6月2日

    Why senior Databricks engineers write less code than mid-level ones

    Two engineers, same team, both five years in. Last quarter Mark shipped forty-seven pull requests across three pipelines. Sam shipped nine. On any dashboard, Mark wins by a mile. Sam got the staff offer. Mark got a kind note about continuing to demonstrate impact. This isn't politics, and it isn't luck. It's a pattern that specifically catches the engineers who are best at shipping, because the most valuable work a senior Databricks data engineer does is invisible by construction. You can't put a ticket number on a problem that never happened. In this episode: - Why the exact behavior that makes you great at mid-level is the behavior that keeps you stuck there - The three categories of senior work that produce zero lines of code but move the entire platform - How to tell leveraged work apart from work that just feels safe to ship - Why "less code" is a symptom and not a goal, and the failure mode of engineers who get that backwards - The one question to ask before your hands hit the keyboard that changes what you volunteer for next sprint This episode is for Databricks data engineers who ship more than anyone on the team and quietly wonder why it isn't landing at review time. Whether you're a mid-level engineer optimizing the wrong line on the chart, or a senior tired of watching your highest-leverage work go uncounted, you'll walk away with language to make prevented work legible and a lens for spending your hours where they actually compound. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

    Why senior Databricks engineers write less code than mid-level ones
查看全部 (17)

關於

Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

資料

你可能也會喜歡