The Microsoft Fabric Pod (by Queryzens)

Query Zens
  • 0.0 (0)
  • Technology

Modern data engineering has gotten unnecessarily complicated. We spend more time connecting separate tools, moving duplicate files, and untangling cloud bills than actually solving interesting problems with data. Welcome to The Microsoft Fabric Pod, brought to you by Queryzens. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier

Episodes

  1. Aug 30

    10 - The Architecture Blueprint & Final Takeaways - The Microsoft Fabric POD

    Welcome to Episode 10—the final episode of our Microsoft Fabric series! Over the past ten episodes, we’ve broken down the whole platform piece by piece. Today, let’s connect all the dots and see the full picture in two minutes: • OneLake: One single home for all your company's data, so teams stop creating isolated folders everywhere. • Delta-Parquet: The open file format that lets Python coders, SQL analysts, and Power BI users work on the exact same data without making duplicate copies. • Shortcuts: Desktop-style pointers that let you query data sitting in Amazon S3 or Google Cloud without spending hours copying files over. • Capacity Units: One single pool of compute power shared across your entire team, cutting down five separate bills into one. • Direct Lake: Lightning-fast Power BI dashboards that read data straight from storage, so nobody has to wait for scheduled dataset refreshes. • Managed Spark: Coding notebooks that start up in seconds, removing all the painful server and cluster setup. • Data Factory: Visual drag-and-drop pipelines and web-based Power Query for quick data cleaning and orchestration. • Real-Time Intelligence: Live event streaming and super-fast KQL queries to handle millions of logs and IoT sensor readings every second. • Purview Governance: Clear permissions, sensitive data tagging, and a visual family tree to trace exactly where every number comes from. When you take a step back, Microsoft Fabric is really just about removing the boring plumbing work—no more endless file transfers, no duct-taping five tools together, and no managing idle servers. You can simply focus on building good data products and helping your business make better decisions. Thank you so much for tuning in to this very first series of The Microsoft Fabric Pod! If this series helped you understand the platform better, please take a moment to subscribe, drop a quick rating on your podcast app, and share the show with your friends and teammates. Until next time, keep your systems simple and happy building! About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    10 - The Architecture Blueprint & Final Takeaways - The Microsoft Fabric POD

  2. Aug 30

    09 - Security, Permissions, and Governance - The Microsoft Fabric POD

    Welcome back to Episode 9! Now, when you put all of a company's files into one single lake, there is one obvious question every IT team asks: How do we make sure people only see what they are supposed to see? Putting all your data in one place definitely does not mean making it open to everyone. Fabric manages security and governance across four simple layers: • Workspace Roles: This is your basic team access. You decide who can build data pipelines, who can run Spark code, and who is just an everyday viewer opening dashboards. • Row and Column Level Security: You can easily restrict specific rows or columns. For example, if a sales lead in Europe runs a SQL query on a global sales table, they will only see European numbers—even though data for India and the US sits in the exact same table in OneLake. • Sensitivity Labels: Fabric connects directly with Microsoft Purview Information Protection. If you mark a table as "Confidential" or "Internal Only," that security label sticks to the data. Even if an analyst exports the numbers into an Excel sheet or builds a Power BI chart, the file remains locked and protected. • Automatic Lineage: Thanks to Microsoft Purview, Fabric draws a visual family tree for your data. If someone spots a strange number on a dashboard, you don't need to ask five people what happened. You can just click the visual map and trace that number backward—through the SQL query, through the Spark cleaning script, all the way to the raw file that arrived in the morning. It gives your IT and security teams complete peace of mind because they can see exactly who is touching what. In Episode 10, we’ll bring all the pieces together and wrap up the entire series. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    09 - Security, Permissions, and Governance - The Microsoft Fabric POD

  3. Aug 30

    08 - Real-Time Intelligence and Streaming Data - The Microsoft Fabric Data

    Welcome back to Episode 8! Today, we're talking about data that simply cannot wait for an overnight batch run: Real-Time Intelligence. Think about cases like live IoT factory sensors, server error logs, user clicks on a shopping app, or payment transactions. If you run a data pipeline only once an hour or once a day, you will find out about problems way too late. You need to see and react to what is happening right now. Fabric manages this live streaming data using two main tools: 1. Eventstreams: This is a simple, no-code way to catch data while it’s moving. You just point a feed from tools like Apache Kafka, Azure Event Hubs, or a webhook directly into Fabric. You can filter out bad rows or clean incoming JSON data on the fly before it even hits the disk. 2. KQL Databases: This uses the Kusto engine—the same tech behind Azure Data Explorer. Standard SQL databases often choke when you dump millions of log entries into them every minute. KQL is designed specifically to scan billions of timestamped rows in less than a second. You can write simple KQL queries, or if you don't know KQL, you can even use regular SQL syntax to search your logs. From there, you have two quick ways to use the data: • Drop the feed directly into a live dashboard so numbers update on your screen in real time. • Use Data Activator (Fabric Activator) to set up automated alerts—for example, sending a Teams ping or triggering a workflow the moment server errors cross a danger limit. And the best part? All of that incoming live data automatically settles into OneLake alongside your normal daily batch tables. In Episode 9, we’ll look at how security, access permissions, and governance work with Microsoft Purview. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    08 - Real-Time Intelligence and Streaming Data - The Microsoft Fabric Data

  4. Aug 30

    07 - Data Factory, Pipelines, and Dataflows Gen2 - The Microsoft Fabric POD

    Welcome back to Episode 7! Today, let’s look at Data Factory inside Fabric and how we bring data into the system. Now, not every team wants to write lines of Python or Spark code just to move files around. Fabric includes simple, visual tools for this, broken into two main parts: 1. Data Pipelines: Think of a pipeline like a project manager for your data. If you need to pull daily sales records from an office SQL server, check if the file exists, run a Spark script to process it, and send a Teams or email alert if anything fails—you build that visually inside a pipeline. It comes with ready-made connectors for databases, APIs, and cloud tools, so you never have to write custom connection code from scratch. 2. Dataflows Gen2: This is essentially Power Query in your web browser. If a business analyst needs to clean messy data—like splitting full names into first and last, removing blank rows, or combining two sheets—they don't need Python. They can just point, click, and transform data right on their screen. The biggest upgrade with Dataflows Gen2 is where that clean data goes. In older setups, cleaned Power Query data was locked inside Power BI. With Gen2 in Fabric, the moment you finish cleaning a table, it writes directly into OneLake as open Delta-Parquet files. That means an analyst can clean a dataset with simple mouse clicks in Power Query, and a data engineer can immediately pick up that exact same table in Python or SQL without any extra export steps. In Episode 8, we’ll move away from scheduled daily batches and look at handling live, real-time streaming data. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    07 - Data Factory, Pipelines, and Dataflows Gen2 - The Microsoft Fabric POD

  5. Aug 30

    06 - Data Engineering with Managed Apache Spark - The Microsoft Fabric POD

    Welcome back to Episode 6! Today, let’s talk about writing code and processing heavy datasets using Apache Spark inside Fabric. If you have ever set up Spark on your own or worked with cloud clusters, you know what a headache it can be: • You have to decide how many worker nodes you need. • You tweak driver memory and runtime versions. • And every time you want to run a simple script, you sit and stare at the screen for five to ten minutes just waiting for the cluster to turn on. Fabric makes Spark completely managed, which removes almost all of that background setup. Here is what changes in your everyday coding: 1. Notebooks start up instantly: Fabric keeps pre-warmed servers running in the background. When you open a Jupyter notebook and hit run, your code starts executing within seconds—no more waiting for clusters to warm up. 2. OneLake is directly connected: You don’t need to write long connection strings, manage secret keys, or mount storage drives. Your OneLake folders appear right in a sidebar. You can literally drag and drop a table name into your notebook cell, and Fabric writes the Python code to read that table automatically. 3. Shared team libraries: If your team needs specific Python packages like Pandas or Scikit-learn, you install them once into a shared environment. You attach that to your workspace, and everyone on the team is immediately on the exact same version. Nobody gets errors like "it works on my machine but not on yours." It takes away all the server babysitting so you can just focus on writing your Python and SQL transformations. In Episode 7, we’ll look at how non-coders and data teams can clean and move data using visual pipelines. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    06 - Data Engineering with Managed Apache Spark - The Microsoft Fabric POD

  6. Aug 30

    05 - Direct Lake Mode in Power BI - The Microsoft Fabric POD

    Welcome back to Episode 5! Today, we’re looking at what is probably the best and most useful feature in Fabric for everyday reporting: Direct Lake Mode in Power BI. If you have ever built large Power BI dashboards, you already know the big trade-off we've had to deal with for years: Import Mode vs. DirectQuery. • With Import Mode: Power BI loads a full copy of your data into its own memory. The reports are blazing fast to click around, but you have to schedule data refreshes. If a refresh fails or takes two hours, your users are stuck looking at yesterday's numbers. Plus, you are paying to store duplicate copies of the same data. • With DirectQuery: Power BI doesn’t copy anything. Every time someone clicks a filter or changes a date on the dashboard, it sends a live SQL query back to your database. The data is always current, but the moment ten people open the dashboard at once, the whole database slows down and reports start lagging. Fabric introduces a third, much better option: Direct Lake Mode. Remember how we said OneLake stores all data in standard Parquet files? Because of that, Power BI doesn’t need to convert or import anything. It reads those clean files straight from OneLake storage directly into memory when a report opens. What does this mean in plain terms? 1. You get the super-fast speed of Import Mode. 2. You never have to wait around for scheduled dataset refreshes. 3. You don't create extra duplicate copies inside Power BI. The second your data pipeline writes new rows into OneLake, Power BI is ready to show them on the very next click. Next up in Episode 6, we’ll look at how data engineers write code using managed Spark in Fabric. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    05 - Direct Lake Mode in Power BI - The Microsoft Fabric POD

  7. Aug 30

    04 - Compute and Capacity Units (CUs) - The Microsoft Fabric POD

    Welcome back to Episode 4! Today, let's talk about compute power and how pricing actually works in Fabric. In traditional cloud setups, managing your bill is painful because you pay for every single tool separately: • You pay for a data pipeline runner in Azure Data Factory. • You pay for a Spark cluster that might be sitting idle. • You pay for an always-on dedicated SQL database. If no one uses that SQL database over the weekend, you still get charged for every single hour unless someone writes a script to shut it down on Friday evening. Fabric fixes this by using something called Capacity Units, or CUs. Instead of renting individual servers for every tool, you buy one shared bucket of computing power for your whole team. Think of it like a shared office Wi-Fi connection: • When your data pipeline runs at 2:00 AM, it uses power from that pool. • When your data engineer writes Spark code at 10:00 AM, it draws from the same pool. • When managers check Power BI reports after lunch, they use that exact same capacity. Everyone shares the same pool, and you only pay one single compute bill. Now, what happens when someone runs a massive job that takes a lot of processing power? Fabric handles this with a feature called smoothing. If a heavy query creates a sudden 5-minute spike in usage, Fabric doesn't immediately crash your system or throw an error. Instead, it spreads out and averages that load over a wider window of time. Your reports keep running, your users don't get blocked, and you don't have to overpay for a massive server just to handle a temporary 5-minute rush. It turns computing power into a simple shared utility, rather than ten different servers you have to babysit. In Episode 5, we’ll look at how this changes Power BI reporting using Direct Lake. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    04 - Compute and Capacity Units (CUs) - The Microsoft Fabric POD

  8. Aug 30

    03 - Shortcuts: Querying External Without Moving Files - The Microsoft Fabric POD

    Welcome back to Episode 3! Today, let's tackle a question almost every team asks: What if our data is not in Microsoft Azure? Let’s be honest—almost no company keeps everything in just one cloud. You might have years of website logs stored in Amazon S3, or your digital marketing team might be saving all their click data in Google Cloud Storage. Usually, if you want to use that data inside Azure, it’s a big headache: • You have to build a pipeline to transfer the data. • AWS or Google charges you network egress fees just to move the files out. • You end up waiting hours for huge copy jobs to finish, creating even more duplicate files. Fabric solves this with a feature called Shortcuts. Think of a shortcut just like a desktop shortcut on your computer. The original file stays right where it is, but you can open and work on it from somewhere else. In Fabric, you simply: 1. Open your workspace. 2. Enter your AWS or Google Cloud credentials. 3. Point directly to that external storage bucket. That’s it. From that moment on, your Spark notebooks and SQL queries see that Amazon S3 folder just like a regular local folder inside Fabric. The files never actually leave AWS or Google Cloud. You don’t have to wait for heavy data copy jobs, and you can write a single SQL query that joins an Azure customer table directly with an Amazon S3 order table. No moving files around, no duplicate storage bills. In Episode 4, we’ll look at how compute power works in Fabric and how Microsoft bills you for running your queries. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    03 - Shortcuts: Querying External Without Moving Files - The Microsoft Fabric POD

  9. Aug 29

    02 - OneLake and Why Open Formats Matter - The Microsoft Fabric POD

    Welcome back to Episode 2! Today, let's talk about the heart of Fabric: OneLake. People often say OneLake is like "OneDrive for your data." It sounds a bit like an ad, but it’s actually the easiest way to picture it. Think about how most offices work today: • The sales team creates their own storage account. • Finance has their own folder somewhere else. • The engineering team has a third one. Now, what happens if finance needs sales data to check company revenue? Someone has to sit and write a script to copy huge files from one team's storage to another. It creates double the files and double the storage cost. OneLake removes this headache. When your company turns on Fabric, you get just one single place for everything. Sales, finance, engineering—every workspace sits inside this one big lake. Now, how does this actually work without everything crashing? Fabric uses an open file format called Delta-Parquet by default. Here is why that matters: • It keeps the files compressed so they don't take up too much space. • It makes sure data doesn't get corrupted if two people use it at once. • Most importantly, it is not a locked Microsoft-only format. It's open-source. So, when your data engineer writes a table using Python or Spark, your SQL analyst can run a query on that exact same file immediately. No one needs to export anything to an Excel sheet. No one needs to run a data copy job. It’s just one single file sitting on the disk, and everyone works on it directly. In Episode 3, we'll see what you do if your company's data is already sitting in AWS or Google Cloud. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    02 - OneLake and Why Open Formats Matter - The Microsoft Fabric POD

  10. Aug 29

    01 - Why Did Microsoft Build Fabric? - The Microsoft Fabric POD

    Hey everyone, welcome to the very first episode of The Microsoft Fabric Pod. Really happy you’re tuning in today. Whether you’re commuting, out for a walk, or just taking a chai break from work, this show is all about explaining data tools in simple words—just like two colleagues talking over coffee. Let’s talk about why Microsoft actually created Fabric. If you have worked on any cloud data project recently, you know what a headache it is. Your whole setup is a mess. Just think about what we usually do: • First, you store raw files in a cloud storage folder. • Then you use Azure Data Factory just to shift those files. • Then you turn on Spark in Databricks or Synapse to clean that data using Python. • After that, you copy the same data into a SQL database so business analysts can run their queries. • And finally, you load it all into Power BI so managers can see their charts. That’s five different software tools just for one job. Every single tool needs its own access permissions, its own network setup, and its own separate billing. And the worst part? Every time you move data from one tool to another, you are making another copy of the same file. It wastes money, wastes time, and becomes impossible to manage. Fabric is Microsoft's way of fixing this mess. Instead of making you connect all these pieces yourself, they put everything in one place—data pipelines, Spark coding, SQL, streaming data, and Power BI. You only need to know two main things: 1. OneLake: Think of it like OneDrive, but for all your big data. You save a file once, and every tool uses that same file. No extra copies. 2. Capacity Units: One single compute bill for everything, instead of paying five separate bills. Over the next ten episodes, we will go through each part step by step and see how it works in real daily work. Let’s jump into Episode 2 and see what OneLake is all about. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier to understand, one conversation at a time.

    01 - Why Did Microsoft Build Fabric? - The Microsoft Fabric POD

  11. Aug 29

    Intro - The Microsoft Fabric POD

    Setting up a data platform these days has honestly become a headache. You buy cloud storage from one service, build your pipelines in another, spin up a Spark cluster somewhere else, manage a separate SQL warehouse, and then spend hours trying to make Power BI read all of it. Half of our working hours just go into moving files back and forth—and paying a separate bill for every single step. That’s why Microsoft came out with Fabric. Their pitch was simple: What if we put all of these tools inside one single screen, and just stopped making duplicate copies of our data? Welcome to The Microsoft Fabric Pod. In this first 10-episode series, we’re going to look at what Fabric actually does behind the scenes—from OneLake and Direct Lake mode in Power BI, to connecting AWS data with shortcuts, running managed Spark, and understanding how the billing really works. No marketing jargon, no buzzwords. Just practical, everyday data engineering. Make sure you hit follow on your podcast app, and let’s jump straight into Episode 1. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZensMaking Data Engineering easier to understand, one conversation at a time.

    Intro - The Microsoft Fabric POD
  • 11 Episodes

About

Modern data engineering has gotten unnecessarily complicated. We spend more time connecting separate tools, moving duplicate files, and untangling cloud bills than actually solving interesting problems with data. Welcome to The Microsoft Fabric Pod, brought to you by Queryzens. About the Host Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms. Hosted by Rajnish Pandey | QueryZens Making Data Engineering easier

Information

You Might Also Like