DevOps & Cloud Interview Prep: Real Scenarios & Answers

DevOpsInterview.Cloud

Real DevOps and cloud interview scenarios, broken down the way a senior engineer would answer them. Kubernetes, AWS, Terraform, CI/CD, observability and incident response, with the trade-offs interviewers actually probe for. More prep material at devopsinterview.cloud.

  1. 6d ago

    AWS Service Control Policies: Prevent Region Sprawl

    A walkthrough of writing AWS Service Control Policies that restrict resource creation to approved regions while carving out exceptions for global services like IAM, CloudFront, and Route 53. Covers the aws:RequestedRegion condition key, common SCP mistakes, and how to answer this question in a DevOps or cloud security interview. Full interview prep guides and scenario walkthroughs: DevOpsInterview.Cloud Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript An audit runs across your AWS Organization and the output is ugly. Resources sitting in fifteen regions nobody approved. EC2 instances in ap-southeast-2, an RDS cluster in sa-east-1, a stray VPC in eu-north-1. Nobody remembers creating most of it. That's region sprawl, and it's one of the most common findings in a cloud security review, because by default AWS lets any account with the right IAM permissions spin up resources in any region on earth. Interviewers ask about this because it tests three things at once. Do you understand the difference between IAM policies and Service Control Policies, or SCPs. Do you know how AWS evaluates deny logic across an Organization. And can you actually write a policy that blocks sprawl without breaking services that are inherently global, like IAM or CloudFront. A lot of candidates can say "use an SCP" but fall apart the moment you ask them to handle the global service exceptions. That's the part that separates people who've read about SCPs from people who've actually deployed one. Here's the mental model. An SCP doesn't grant any permissions. It's a guardrail, not a grant. Every account in your Organization already gets whatever IAM policies and roles say it can do. The SCP sits above that, at the Organization or Organizational Unit level, and it filters the maximum available permissions. Think of IAM as the accelerator and SCP as a speed limiter bolted onto the engine. If the SCP denies an action, it doesn't matter what the IAM policy says, even a full admin role can't override an SCP denial. That's why SCPs are the right tool for org-wide guardrails like region restriction, instead of trying to chase down every IAM policy in every account. The core mechanism is a condition key called aws colon RequestedRegion. You write a deny statement with a wildcard action, a wildcard resource, and a condition that says: if the requested region is not in my approved list, deny everything. Something like this. Effect deny, action star, resource star, condition StringNotEquals on aws colon RequestedRegion, with a list of your approved regions, say us-east-1, us-west-2, and eu-west-1 if you're a US and UK facing company. Attach that SCP to the root of the Organization, or to a specific Organizational Unit if you want different rules per business unit, and now any API call targeting a region outside that list gets an explicit deny, no matter which account or which IAM role makes the call. But if you stop there, you break your own Organization. A chunk of AWS services are global and don't have a regional home the way EC2 or RDS do. IAM itself is global. Organizations is global. Route 53 is global. CloudFront is global, and if you're using CloudFront with a custom certificate, that certificate has to live in ACM in us-east-1 specifically, regardless of where your users are. AWS Support, AWS Billing, AWS WAF when attached to CloudFront, and AWS Shield also operate globally or default to us-east-1 under the hood. If your deny statement doesn't carve these out, you'll lock yourself out of managing users, you'll break certificate issuance for your content delivery network, and you'll have people filing tickets within the hour. The fix is to add a NotAction clause, or layer a second condition, that excludes the actions belonging to those global services from the region check. In practice this looks like a list of service prefixes you exclude: iam colon star, organizations colon star, route53 colon star, cloudfront colon star, support colon star, and the global billing and cost management actions. You put those under NotAction in the same deny statement, so the statement reads: deny everything, on every resource, when the region isn't approved, except for these specific global service actions. That's the version that actually survives contact with a real Organization. Let's walk through the logic step by step the way you'd explain it on a whiteboard. First, decide your approved region list based on where your actual customers and compliance requirements sit. For a company serving the US and UK, that might be us-east-1 for primary workloads, us-west-2 for disaster recovery, and eu-west-1 for UK and EU data residency. Second, write the deny statement with action star and resource star, scoped by the RequestedRegion condition using StringNotEquals against that list. Third, add the NotAction exclusion list for IAM, Organizations, Route 53, CloudFront, Support, Trusted Advisor, and the Billing console actions, because none of those are meaningfully regional and blocking them breaks account administration, not just one workload. Fourth, attach the SCP at the right level. Attaching it to the Organization root affects every account including your management account, which is usually too aggressive. Most teams attach it to a workload Organizational Unit that holds production and staging accounts, and keep the management account and any security tooling accounts in a separate OU with lighter restrictions. There's a nuance interviewers like to probe: SCPs apply to everyone in the account except the Organization's management account itself, which SCPs cannot restrict. So if someone asks "what happens if the management account creates resources in ap-south-1," the honest answer is that SCPs won't stop it, you need separate controls like AWS Config rules or Control Tower guardrails for that account specifically. Another nuance: SCPs are evaluated alongside IAM policies, resource policies, and permission boundaries, and an explicit deny anywhere in that chain wins. So even if someone later attaches a broad IAM policy allowing EC2 everywhere, the SCP's deny on RequestedRegion still blocks it. That's the enforcement guarantee that makes SCPs worth using over just writing style guides and hoping people follow them. Now the common wrong answers, because this is where candidates lose points even when they know the general idea. Wrong answer one: writing the SCP as an Allow statement listing only the approved regions, instead of a Deny with NotAction exclusions. Allow-only SCPs are dangerous because you have to enumerate every single service and action you ever want permitted, and you'll inevitably miss something, usually a global service, and break production. Deny with explicit carve-outs is safer and easier to reason about. Wrong answer two: forgetting that S3 bucket names and some API calls are global in namespace even though the bucket itself is regional, which trips people up when they test the policy and S3 behaves unexpectedly. Wrong answer three: attaching the SCP to the entire Organization including the management account and sandbox or security OUs, which blocks legitimate cross-region testing and incident response work. Wrong answer four: not testing the SCP in a non-production OU first. SCPs are unforgiving, a typo in the condition key, like getting the case wrong on RequestedRegion, silently fails to apply the restriction at all, so you think you're protected when you're not. Always validate with the IAM policy simulator or a controlled test account before rolling it out organization-wide. One more thing worth mentioning if the interview goes deeper: this pairs well with AWS Control Tower, which ships pre-built region-restriction guardrails you can enable instead of hand-rolling the policy, and with AWS Config aggregators that give you the visibility to actually catch region sprawl in the first place, since an SCP prevents new sprawl but doesn't clean up the fifteen regions you already found in the audit. For that, you still need a remediation pass, tagging what's legitimate, terminating what isn't, and documenting exceptions like a DR region or a specific compliance region for UK data residency under something like the UK's version of data protection requirements. Quick recap. Region sprawl happens because AWS doesn't restrict regions by default and IAM alone can't enforce it organization-wide. SCPs sit above IAM and enforce a hard ceiling on permissions across every account in an OU. The core tool is a Deny statement using the aws colon RequestedRegion condition key, scoped to your approved regions, with a NotAction exclusion list for global services like IAM, Organizations, Route 53, CloudFront, and Support. Attach it at the right OU level, never expect it to restrict the management account, and always test before rolling out, because a misconfigured condition key fails silently. Get this right and you'll sound like someone who's actually run an AWS Organization, not someone reciting a service page. If you want the full written version of this with the complete policy JSON, the global service exclusion list, and the exact OU structure, head to devopsinterview dot cloud and grab the prep guide. It's laid out so you can read it the night before the interview and walk in with the details straight.

    AWS Service Control Policies: Prevent Region Sprawl

  2. Oct 31

    SQS Dead Letter Queue Storm: Diagnose and Recover

    You wake up to 10,000 messages in your SQS dead letter queue — here's the systematic CloudWatch-driven approach to diagnosing the root cause and safely replaying messages without triggering a second storm. You'll learn: Which four CloudWatch metrics to pull first (ApproximateNumberOfMessagesVisible, NumberOfMessagesReceived, NumberOfMessagesDeleted, ApproximateAgeOfOldestMessage) and how to read the curve shape to distinguish a deploy failure from resource exhaustionWhy redriving immediately is the most common wrong answer — and how to confirm the root cause is actually resolved before touching the DLQHow visibility timeout math interacts with Lambda timeouts to silently manufacture poison pill storms at scaleUsing StartMessageMoveTask (or the console's Start DLQ redrive) for rate-controlled replay instead of a manual receive-delete-republish loopHow to set maxReceiveCount deliberately — why 1-2 is too aggressive and 10 is too lax, and how to align visibility timeout to your p99 processing timeKeywords: SQS dead letter queue, DLQ redrive policy, SQS visibility timeout, AWS SQS CloudWatch metrics, SRE interview questions AWS 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript You wake up to a Slack alert. Your dead letter queue, the DLQ, has ten thousand messages sitting in it. Somewhere overnight, a queue that normally holds zero items quietly filled up, and nobody got paged until the number crossed a threshold. This is one of those incidents that looks small until you realize it might mean ten thousand failed payments, ten thousand missed notifications, or ten thousand orders stuck in limbo. Interviewers love this scenario because it tests three things at once. Can you read CloudWatch metrics fast enough to find the actual root cause instead of guessing. Do you understand how a redrive policy and visibility timeout interact. And can you recover the messages without making the outage worse by replaying them straight into the same failure loop. Start with the mental model. In SQS, every queue can have a redrive policy pointing to a dead letter queue, along with a maxReceiveCount. Every time a consumer pulls a message and fails to delete it, either by crashing, throwing an error, or just running past the visibility timeout, the receive count on that message goes up by one. Once it crosses maxReceiveCount, SQS automatically moves that message to the DLQ. The DLQ is not where messages go when something is broken right now, it is where they go after they have already failed a set number of times. So by the time you see ten thousand messages there, the failures happened earlier and repeated, meaning you are looking at an aftermath, not a live fire. First move: resist the instinct to immediately redrive the messages back to the source queue. That is the single most common wrong answer, and it will just refill the DLQ again if the underlying cause is still active. Diagnose first. Open CloudWatch and pull four metrics for the source queue: ApproximateNumberOfMessagesVisible, NumberOfMessagesReceived, NumberOfMessagesDeleted, and ApproximateAgeOfOldestMessage. Then pull ApproximateNumberOfMessagesVisible for the DLQ itself, graphed over the last twelve hours. You are looking for the shape of the curve. A sudden vertical jump around a specific timestamp tells you something changed at that moment, like a deploy, a dependency outage, or a config push. A slow staircase climbing over hours usually points to a resource exhaustion problem, like a database connection pool filling up under load. Next, check your consumer's own logs or traces around that timestamp. In a Lambda-based consumer, look at Errors and Throttles in CloudWatch for the function, plus Duration against your function timeout. A classic failure pattern: someone set the Lambda timeout to three seconds, a downstream API started taking four seconds under load, every invocation started timing out, SQS saw the message never get deleted, the visibility timeout expired, the message came back, failed again, and after three or five retries it landed in the DLQ. Multiply that by every message hitting the same slow endpoint and you get your ten thousand. Another classic: a single malformed message, a poison pill, causes a parsing exception on every attempt. But a poison pill alone usually only produces one or two hundred failures before someone notices, not ten thousand, unless the queue has heavy throughput. If you see ten thousand nearly identical failures with the same exception type and the same payload shape, that is a code or schema problem, not a flaky dependency. If you see varied payloads but a shared downstream timeout pattern, that is capacity, not code. Once you know the category, check whether it is still happening. Look at the error rate for the last fifteen minutes, not twelve hours ago. If errors are still climbing, fix the root cause before you touch the DLQ at all. If the downstream service is still degraded, redriving now just recreates the storm. If the root cause is already resolved, like a deploy that got rolled back, you can move to recovery. For recovery, the safe method is a redrive using the SQS console's Start DLQ redrive feature, or the StartMessageMoveTask API, which lets you move messages from the DLQ back to the source queue at a controlled rate rather than all at once. Do not write a custom loop that receives all ten thousand messages and deletes and republishes them manually, that approach risks duplicate processing and loses the built in rate limiting. If you need custom filtering first, like removing a known bad payload format, consume a sample batch, inspect the pattern, and only then decide whether to redrive the whole batch or quarantine a subset. Watch NumberOfMessagesSent and NumberOfMessagesDeleted on the source queue during the redrive. If deletes keep pace with sends, your fix worked and the queue is draining cleanly. If visible message count on the DLQ starts climbing again within minutes, your root cause fix did not fully work, stop the redrive immediately, you just prevented storm number two. Now design the redrive policy properly so this does not repeat. First, pick maxReceiveCount deliberately. A value of one or two is too aggressive for transient network blips, you will dead letter messages that would have succeeded on a third try. A value of ten is usually too lax, you will mask a real problem for a long time before anyone notices. Three to five is a reasonable default for most workloads, tuned against how expensive a false negative is for your business case. Second, align your visibility timeout with your actual processing time, not a guess. If your consumer normally finishes in two seconds but occasionally spikes to eight under load, set the visibility timeout to something like six times your p99 processing time, so a message does not become visible again and get double-processed while the original attempt is still legitimately running. AWS's own guidance is to set it to at least the function timeout, but in practice you want headroom for retries inside your own code too. Third, add alerting on the DLQ itself, specifically ApproximateNumberOfMessagesVisible greater than zero sustained for more than five minutes. A healthy DLQ should be empty almost all the time. Alerting at zero crossing to any positive number, rather than waiting for ten thousand, is what turns this into a five-minute fix instead of an overnight pile-up. Fourth, consider separating poison pill handling from capacity handling. Some teams add a lightweight validation step before the message even hits the main processing logic, so malformed payloads get dead lettered on attempt one with a clear tag, while legitimate messages that fail due to downstream timeouts get the full retry budget. This avoids wasting retries on messages that will never succeed no matter how many times you try. Common wrong answers interviewers listen for: saying you would just increase maxReceiveCount to a large number to stop messages from dead lettering, which just delays detection and increases duplicate processing risk. Saying you would manually delete the DLQ messages to clear the alert, which destroys the evidence and often the actual customer data you need to reconcile. And redriving before confirming the root cause is fixed, which is the fastest way to turn one bad night into two. Quick recap. A full DLQ is a symptom of a cause that happened earlier and repeated past your maxReceiveCount. Use CloudWatch's visible message count, receive and delete counts, and age of oldest message to find when and how the failures started. Diagnose whether it is a poison pill or a capacity problem before you redrive anything. Use the built-in redrive task for controlled recovery, and tune maxReceiveCount, visibility timeout, and DLQ alerting so the next ten thousand messages never pile up unnoticed. If you want to drill questions like this one until the answers are automatic, head over to the DevOps Interview Cloud channel on YouTube for daily thirty-second interview drills, and check devopsinterview dot cloud for the full breakdown of this scenario.

    SQS Dead Letter Queue Storm: Diagnose and Recover

  3. Oct 28

    RDS Proxy: Fixing Lambda Connection Exhaustion

    RDS Proxy solves the Postgres connection exhaustion that hits when Lambda bursts from 10 to 2,000 concurrent invocations faster than your database can fork new OS processes. You'll learn: Why Lambda's parallelism model collides with Postgres's per-connection process model — and why this is a common senior interview trapHow RDS Proxy multiplexes thousands of client-side Lambda connections onto a small warm pool of real backend connectionsExactly what causes pinning — session variables, temp tables, advisory locks, multi-statement transactions, and some prepared statement patterns — and how pinning defeats pooling entirelyHow to set max connections percent and max idle connections percent to leave headroom for non-proxy clients like migration jobsWhy connection borrow timeout should be kept short so failures surface as fast, retriable errors rather than Lambda timeouts holding resourcesKeywords: RDS Proxy interview questions, Lambda database connection pooling, Postgres connection exhaustion serverless, RDS Proxy pinning causes, Aurora Lambda architecture 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript Picture a Lambda function behind an API Gateway, and it spikes from ten concurrent invocations to two thousand in under a minute. Every one of those invocations opens its own connection to a Postgres RDS instance. Within seconds, Postgres hits its max connections limit, new invocations start throwing connection refused errors, and your error rate graph goes vertical. Nothing about your code changed. Traffic just moved faster than your database could handle new sessions. Interviewers ask about this because it's a trap that catches people who understand Lambda and people who understand databases separately, but not the seam between them. Lambda scales by running many isolated execution environments in parallel, and each one that needs a database connection opens a fresh TCP connection and does a Postgres handshake. Postgres, unlike a typical application server, forks a full OS process per connection, which is expensive in memory and CPU. That's why Postgres defaults usually cap out around one hundred connections, sometimes a few hundred on larger instance classes. Lambda doesn't know or care about that number. You get a mismatch between a scale-to-zero compute model and a stateful, connection-limited database. The mental model here is pooling versus parallelism. Lambda gives you parallelism almost for free. Postgres gives you a hard ceiling on parallel connections. You need something in between that absorbs the burst and smooths it into a connection count the database can sustain. That's the job of RDS Proxy. It sits between your Lambda functions and your RDS or Aurora instance, maintains a warm pool of real database connections, and lets many client-side connections from Lambda multiplex onto a much smaller number of actual backend connections. Lambda thinks it's getting its own connection. Underneath, the proxy is reusing a handful of real ones. Here's how it works. Your Lambda function talks to the proxy endpoint, not Postgres directly. The proxy accepts that connection cheaply because it already holds a pool of established, authenticated backend connections. When your function sends a query, the proxy borrows a pooled backend connection, runs the query, returns the result, and hands that connection back to the pool once your logical unit of work is done. So two thousand invocations each making a quick query might only need twenty or thirty actual Postgres connections, because the proxy cycles them fast. Now the part that actually gets tested: pinning. Pinning happens when the proxy can no longer safely multiplex a client connection onto different backend connections, so it locks that client to one specific backend connection for the rest of its session. Once pinned, you lose the pooling benefit for that session entirely, and if enough sessions pin at once, you're back to the same exhaustion problem you were trying to avoid. Know these triggers cold. Session-level state, like session variables, temp tables, or a changed search path, forces a pin because the proxy can't guarantee that state exists on a different backend next time. Explicit multi-statement transactions pin you to one backend until the transaction resolves. Session-scoped functions and settings, like changing client encoding or using advisory locks, pin because they're tied to a specific backend session. And prepared statements, depending on driver behavior, can pin because the statement lives on a specific backend connection. The practical fix is mostly behavioral. Keep Lambda database interactions short and stateless: open a connection, run your query or a tightly scoped transaction, close it, done. Avoid session-level variables, set things like statement timeout per-query instead of per-session, and commit promptly rather than holding a transaction open across unrelated work. Now the sizing knobs, because candidates often hand-wave here and interviewers notice. Max connections percent is a setting on RDS Proxy that caps what percentage of the database's max connections the proxy is allowed to use. If Postgres allows two hundred connections and you set this to fifty, the proxy caps itself at a hundred backend connections, leaving headroom for other clients like a migration job or an analytics tool connecting directly. If the proxy claims every connection, anything outside the proxy gets locked out. Max idle connections percent controls how many backend connections the proxy keeps warm versus closing when traffic drops. Too low, and every burst after a quiet period pays the cost of re-establishing connections, hurting latency. Too high, and you're holding connections Postgres could serve elsewhere. A reasonable starting point for bursty Lambda workloads: keep max connections percent low enough to leave twenty to thirty percent of the database's slots free for non-proxy use, and set idle percent at roughly half of that max so the pool shrinks during quiet periods without flatlining. There's also connection borrow timeout, how long a client waits for the proxy to hand it a connection before giving up. Keep this short enough that failed requests fail fast and surface as retriable errors, rather than your Lambda function timing out while holding resources. Quick numbers: say your Postgres instance supports two hundred max connections, Lambda runs with reserved concurrency of five hundred, and each invocation does one quick read averaging twenty milliseconds. Without a proxy, five hundred concurrent invocations against a two-hundred-connection ceiling means instant exhaustion. With the proxy capped at a hundred connections, each query only holds a backend connection for twenty milliseconds, so the pool turns over fast enough to comfortably serve far more than a hundred concurrent logical requests, as long as queries stay short and nothing pins. Common wrong answers. First, just raising RDS max connections to some huge number. That doesn't scale, because per-connection memory overhead means you'll run out of RAM or CPU long before hitting some arbitrary ceiling, and you still haven't fixed the architectural mismatch. Second, bundling a generic connection pooler like PgBouncer inside the Lambda deployment package itself. That fails because Lambda execution environments are ephemeral and isolated, so a pool inside one environment can't be shared with another, giving you one pool per invocation and zero pooling benefit. Third, assuming RDS Proxy is a silver bullet that needs zero code changes. If your application is heavy on session state or long transactions, the proxy will dutifully pin almost every connection, and you've paid for a feature you're not using. So here's the thirty-second recap. Lambda scales fast and opens one connection per execution environment. Postgres has a hard connection ceiling because it's process-per-connection. RDS Proxy sits in between, pools backend connections, and multiplexes many Lambda-side connections onto a small number of real ones. Pinning happens when your code uses session state, long transactions, or session-scoped settings, and it defeats the pooling by locking a client to one backend connection. Tune max connections percent to cap the proxy's footprint and leave room for other clients, tune idle percent to balance warm pool size against resource usage, and keep your database interactions short and stateless to avoid pinning in the first place. If you want to drill this kind of scenario until the pinning conditions and the tuning tradeoffs are second nature, head over to the DevOps Interview Cloud channel on YouTube, where we post thirty-second interview drills every day. And if you want the deeper write-ups to go with them, devopsinterview dot cloud has you covered.

    RDS Proxy: Fixing Lambda Connection Exhaustion

  4. Sep 16

    EKS Control Plane Logs: Audit Trail for a Compliance Breach

    A walkthrough of EKS control plane logging for security investigations: which of the five log types actually capture who did what, how to write CloudWatch Logs Insights queries against audit logs, and how to build a metric filter alert on sensitive verbs like delete and create against secrets, roles, and the aws-auth configmap. Visit https://devopsinterview.cloud for interview prep ebooks, roadmaps and hands-on incident labs. Transcript A compliance audit flags a handful of API calls on your EKS cluster that nobody can explain. Someone created a cluster role binding last month. Someone else deleted a secret three weeks ago. The security team wants a name, a timestamp, and a source IP, and you go looking for the trail only to find that nobody turned on the logging that would have captured any of it. That gap is exactly what this question is testing, and it comes up constantly in SRE and platform interviews because it sits at the intersection of Kubernetes internals, AWS billing tradeoffs, and actual incident response. Interviewers ask this because almost everyone can say the phrase enable control plane logging, but very few can explain which of the five log types actually gives you an audit trail versus which ones are just operational noise. It also tests whether you understand that CloudTrail and Kubernetes audit logs are answering two completely different questions, which is a mistake that trips up even experienced engineers. Here's the mental model. EKS gives you five control plane log types: api server, audit, authenticator, controller manager, and scheduler. The controller manager and scheduler logs are mostly useful for debugging cluster behavior, like why a deployment isn't scaling or why pods are stuck pending. They're not what you want for a security investigation. The api server log shows you raw requests hitting the API server, which is useful but incomplete on its own. The two log types that actually matter for a compliance breach are audit and authenticator. The audit log is the Kubernetes API server's own audit trail. Every request that mutates or reads cluster state gets recorded with the verb, like get, create, update, patch, or delete, the object reference, meaning the resource type and namespace, the user identity, the source IP, the user agent, and a timestamp. This is your who-did-what record. The authenticator log is specific to EKS and shows you how IAM identities get mapped into Kubernetes RBAC through the aws-auth configmap or access entries. If someone's IAM role suddenly started acting like cluster-admin, the authenticator log is where you see that mapping happen. A critical point that trips people up in interviews: CloudTrail does not give you this. CloudTrail logs calls to the EKS control plane itself, things like CreateCluster, UpdateNodegroupConfig, or DescribeCluster. It has no visibility into what happens once a request reaches the Kubernetes API server inside the cluster. If someone uses kubectl to delete a secret or create a role binding, that's a Kubernetes API call, not an AWS API call, and CloudTrail simply won't see it. This distinction alone is a good signal in an interview because it shows you actually understand where the AWS control plane ends and the Kubernetes control plane begins. Now the walkthrough. First you enable the logging. Using the AWS command line, you'd run something like aws eks update-cluster-config, specifying your cluster name and a logging block that turns on the api, audit, and authenticator types, and explicitly turns off controller manager and scheduler if you don't need them. Once that's live, EKS starts streaming logs into a CloudWatch log group named something like slash aws slash eks slash your-cluster-name slash cluster, with separate log streams per type. Here's where the cost conversation matters, because interviewers love asking about this tradeoff. A moderately busy cluster, say fifty nodes with a healthy mix of workloads and controllers reconciling state, can easily generate two to five gigabytes of audit log data per day. CloudWatch charges roughly fifty cents per gigabyte ingested, plus storage costs on top, and audit logs are chatty by nature because every single API interaction, including read-only list and watch calls from system components, gets recorded. If you don't set a retention policy on that log group, you're paying to store that indefinitely. A sensible default is something like ninety days retention for audit logs to satisfy most compliance frameworks like SOC 2 or PCI DSS, then export older data to S3 if you need longer retention for legal reasons. Once the logs are flowing, you go to CloudWatch Logs Insights to actually investigate. This is a query language, not full text search, and it's worth knowing the syntax cold for an interview. A basic investigative query looks like this: you select fields for timestamp, the username under user, the verb, the object reference resource, the object reference namespace, and source IPs. Then you filter where verb is in the list delete, create, update, and where the object reference resource equals secrets. Then you sort by timestamp descending and limit the results. That single query answers the exact question compliance asked: who touched secrets, when, and from where. You'd run a similar query filtering on object reference resource equals clusterrolebindings or rolebindings to catch privilege escalation attempts, because creating a binding to the cluster-admin role is one of the most dangerous single actions someone can take in a cluster. You'd also want a query that filters specifically for changes to the aws-auth configmap, since that's the bridge between IAM and RBAC, and any unexpected edit there deserves immediate attention regardless of who made it. Investigation is reactive though, and the real value in an interview answer is showing you'd build proactive alerting instead of waiting for an auditor to ask questions. The pattern here is a CloudWatch metric filter on the audit log group that matches your sensitive verb and resource combination, for example delete against secrets, or create against clusterrolebindings, or any write verb against the aws-auth configmap specifically. That metric filter increments a custom metric whenever a match occurs. You attach a CloudWatch alarm to that metric with a threshold of one event over a five-minute window, and the alarm publishes to an SNS topic that fans out to Slack or PagerDuty. For teams doing more sophisticated response, you route the same metric filter through EventBridge into a Lambda function that automatically pulls the full audit record, checks the source IP against your known CIDR ranges, and opens a ticket automatically if it's an unrecognized source. Let's talk about the common wrong answers, because these show up constantly. The first is enabling only the api server log and assuming that's sufficient, when it actually gives you less structured identity information than the dedicated audit log. The second is treating CloudTrail as the audit trail for in-cluster activity, which as we covered, it simply isn't. The third is enabling everything, all five log types, without setting retention or filtering, which quietly turns into a massive CloudWatch bill within a month with no one noticing until finance asks questions. The fourth mistake is writing alerts that fire on every single write operation, which drowns your team in noise from legitimate CI/CD pipelines doing routine deployments, and within two weeks everyone starts ignoring the alert channel entirely. Good alerting here means scoping tightly to specific verb and resource combinations that are rare and high-risk, not broad categories. One more nuance worth mentioning if the interviewer pushes further: audit log entries include a stage field showing whether the record is the request received, the response started, or the response complete. For investigations you generally want the response complete stage, because that tells you whether the mutating action actually succeeded, versus a request that was received but denied by RBAC. Filtering only on request received would falsely flag denied attempts as successful actions, which matters a lot when you're trying to figure out whether an actual breach occurred or someone just got a permission denied error. So to recap. EKS gives you five control plane log types, but audit and authenticator are the ones that answer compliance and security questions, while controller manager and scheduler are for operational debugging. CloudTrail covers the AWS-level EKS API, not Kubernetes API calls inside the cluster, so don't confuse the two in an interview answer. Enable logging through cluster config, set a sane retention period to control cost, and use CloudWatch Logs Insights queries filtering on verb, object reference resource, and username to trace exactly who did what. Then move from reactive investigation to proactive alerting with metric filters on sensitive verb and resource combinations, especially around secrets, role bindings, and the aws-auth configmap, routed through SNS or EventBridge so you catch the next incident in minutes instead of during next quarter's audit. If you want to drill this kind of scenario until the answer comes out clean under pressure, head to the DevOps Interview Cloud channel on YouTube for daily thirty-second interview drills, and check devopsinterview dot cloud for the full write-up on this one.

    EKS Control Plane Logs: Audit Trail for a Compliance Breach

  5. Sep 9

    AWS Network Firewall: Stop Exfil Without Breaking Egress

    AWS Network Firewall in a centralized inspection VPC is the right answer when security hands you a ticket to block unexpected egress — here's how to design it without taking down production at 2am. You'll learn: Why the firewall lives in one centralized inspection VPC behind AWS Transit Gateway, not deployed per spoke VPC — and how the route tables enforce that How stateful domain-list rules use TLS SNI to allow *.github.com and *.pypi.org while dropping everything else on 443 — without decrypting traffic The encrypted client hello (ECH) blind spot: why SNI-based rules can silently stop matching and what you need as a backstop How to roll out in alert/count mode first, capture a full week of egress patterns to CloudWatch, and flip rule groups to drop one at a time Pricing gotcha: endpoint hours per AZ plus data processed means centralized inspection is cheaper, but you must size for combined peak across all VPCs Keywords: AWS Network Firewall interview questions, TLS SNI egress filtering, centralized inspection VPC, cloud security engineer interview, data exfiltration AWS 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript Security just sent you a ticket. Block all unexpected egress across every VPC. No more instances calling out to random IPs on the internet. Sounds simple until you realize your build servers pull from package registries, your app calls three SaaS APIs, and nobody has a full list of what's actually supposed to be leaving the network. Ship the wrong rule set and you either miss real exfiltration paths, or you break a deploy at two in the morning and get paged for it. This question shows up in senior DevOps and cloud security interviews because it tests three things at once. Can you design for centralized inspection instead of bolting a firewall onto every VPC. Do you understand the tradeoff between IP-based and domain-based filtering. And do you have a rollout process that doesn't nuke production traffic on day one. Anyone can say "add a firewall." Fewer people can explain why it goes in one place, and how you prove it's safe before you flip it to blocking mode. Start with the mental model. In a multi-VPC setup, you do not want AWS Network Firewall deployed separately in every spoke VPC. That's operationally painful, it's expensive, and every team ends up writing slightly different rules. The standard pattern is a centralized inspection VPC, sitting between your spoke VPCs and the internet, usually paired with AWS Transit Gateway. Every spoke VPC routes its default egress through the transit gateway to the inspection VPC. The firewall endpoints live there, traffic gets inspected once, using one shared rule set, and then it's routed out through a NAT gateway to the internet. The firewall itself has two kinds of rules. Stateless rules operate on packet headers, source and destination IP, ports, protocol. They're fast, but they don't know anything about the content or the destination domain. Stateful rules track connections and can inspect protocol details, and this is where domain filtering comes in. AWS Network Firewall supports domain-based stateful rules using the HOME_NET and TLS SNI, the Server Name Indication field. Since most traffic is encrypted, the firewall reads the SNI in the TLS handshake to figure out what domain a connection is headed to, without decrypting the actual payload. Here's the walkthrough security actually wants. Instead of writing rules like "allow this IP range," which breaks the moment a SaaS vendor rotates their infrastructure behind a content delivery network, you write domain allow lists. Something like: allow egress to star dot amazonaws dot com, star dot github dot com, star dot pypi dot org, your internal artifact registry domain, and your approved SaaS endpoints like star dot datadoghq dot com. Everything else on port 443 and port 80 gets dropped by a default deny rule at the bottom of the rule group. That's the actual control that stops exfiltration. An attacker who compromises a workload can't just open a connection to some random domain and quietly ship data out, because the SNI doesn't match anything on the allow list and the connection gets reset. A concrete example: say you've got an application VPC running ECS tasks that call out to a payments API and pull container images from Amazon ECR. You'd build a stateful rule group with domain list rules, action equals pass, targeting star dot amazonaws dot com for ECR and S3 access, plus the specific payments vendor domain. Then a second rule group, lower priority, with a drop-all rule for anything else on 443. You attach both rule groups to a firewall policy, and that policy gets applied to the firewall in the inspection VPC. Every spoke VPC's route table sends 0.0.0.0/0 traffic to the transit gateway attachment for the inspection VPC, so nothing bypasses the firewall. Now the tradeoffs, because this is where people get burned. Domain filtering by SNI only works for standard TLS on the usual ports, and only if the client isn't using something like encrypted client hello, which hides the SNI entirely. If a vendor rolls out ECH, your domain rule silently stops matching and either everything from them gets blocked, or worse, it falls through some overly broad rule you added while troubleshooting. You also can't domain-filter plain HTTP or arbitrary custom ports the same way, so you still need IP or port-based stateless rules as a backstop for non-standard traffic, like a database replication port or a syslog forwarder hitting a fixed IP. Cost and scale matter too. Network Firewall pricing is based on endpoint hours per availability zone plus data processed, so centralizing it in one inspection VPC is cheaper than deploying it per spoke VPC, but you're also creating a single choke point. Size your endpoints for peak traffic across all VPCs combined, not just one team's workload, or you'll see latency spikes during deploys when everyone's pulling images at once. The rollout process is the part that saves you from an outage. Never go straight to a default-deny domain policy in a live account. Start in alert mode, sometimes called count mode, where the firewall logs what it would have blocked without actually dropping it. Ship those logs to CloudWatch or S3, run them for at least a full business cycle, ideally including a deploy day, a patch day, and a batch job day, because those all have different egress patterns. Build your domain allow list from that real traffic, not from what people think they're calling. Then flip specific rule groups from alert to drop one at a time, watch error rates and application logs, and keep a documented rollback, which just means removing that rule group from the policy or reverting the route table change on the spoke VPC. Common wrong answers you'll hear in interviews. One: "just use security groups." Security groups are IP and port based, they don't understand domains, and CDN-fronted SaaS traffic breaks them constantly. Two: "deploy Network Firewall in every VPC." Technically works, but it's expensive, hard to keep consistent, and it's not what "centralized inspection" architecture means when an interviewer asks for it. Three: "block everything by IP range from threat intel feeds." That catches known-bad infrastructure but does nothing against a compromised workload calling out to a brand-new domain that hasn't been flagged yet, which is exactly the exfiltration scenario security is worried about. Four: forgetting DNS. If you don't also control DNS resolution, or at least log it through Route 53 Resolver query logging, attackers can tunnel data through DNS queries themselves, which a domain-based TLS rule won't ever see. Quick recap. Centralize AWS Network Firewall in a dedicated inspection VPC, route all spoke VPC egress through transit gateway into it. Use stateful domain-based rules keyed off TLS SNI for an allow list of legitimate destinations, backed by stateless IP and port rules for non-TLS or fixed-port traffic. Roll out in alert mode first, build your allow list from real traffic patterns across a full business cycle, then convert to drop mode one rule group at a time. Watch for encrypted client hello breaking SNI matching, and don't forget DNS query logging as a second exfiltration channel outside the firewall's visibility. If you want the full written version of this, with the actual rule group syntax and a rollout checklist you can bring into an interview or a real migration, it's all laid out over at devopsinterview dot cloud.

    AWS Network Firewall: Stop Exfil Without Breaking Egress

  6. Aug 30

    CoreDNS at 100K RPS: ndots, Negative Caching, and Autopath

    A real-world CoreDNS latency incident at high query volume reveals how ndots and Kubernetes search domains silently multiply DNS lookups into a full-blown query storm. You'll learn: Why a single external hostname like api.stripe.com generates 4 upstream queries under Kubernetes' default ndots:5 — and how that 4x amplifier compounds at scale How to override search behavior per-pod via dnsConfig.ndots, and why a trailing dot in your FQDN collapses four queries into one Tuning the CoreDNS cache plugin: setting denial TTLs high enough that NXDOMAIN responses actually stick, and the staleness tradeoff you must articulate in interviews What the autopath plugin actually does differently — moving search-path resolution server-side — and when it helps versus when it shifts the problem Why interviewers use this question to separate engineers who've read the docs from those who've debugged a DNS storm at 3am Keywords: CoreDNS tuning, Kubernetes DNS ndots, negative caching NXDOMAIN, CoreDNS cache plugin, Kubernetes networking interview 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript Picture this. Your cluster is doing something like a hundred thousand requests per second across a few hundred services, and suddenly every outbound call to a third-party API starts timing out intermittently. Nothing in your application changed. No deploy went out. But your p99 latency for anything that touches DNS resolution just tripled, and your CoreDNS pods are pegged at max CPU. This is a real failure pattern, and it almost always traces back to two things: how ndots is configured in every pod's resolver, and how CoreDNS is caching, or failing to cache, negative responses. Interviewers ask about this because it separates people who've read the Kubernetes docs from people who've actually had to explain a production DNS storm to their team. Anyone can say 'CoreDNS handles DNS in the cluster.' Far fewer people can explain why a single hostname lookup can turn into four or five actual queries hitting your DNS servers, and what that does to your query volume when you're already running near capacity. This question tests whether you understand resolution mechanics, not just that a service called CoreDNS exists. Here's the mental model. Every pod in a Kubernetes cluster gets a resolv.conf file, and that file has two important settings: a search list and an ndots value. The search list typically looks like namespace dot svc dot cluster dot local, then svc dot cluster dot local, then cluster dot local, and depending on your cloud provider, maybe an additional domain from the node itself. The ndots value, five by default in Kubernetes, tells the resolver: if the name you're looking up has fewer than five dots in it, don't treat it as fully qualified. Instead, try appending each entry in the search list first, in order, before falling back to treating it as an absolute name. Now think about what that means for a completely ordinary external lookup, something like api dot stripe dot com. That name has two dots. Since two is less than the ndots threshold of five, the resolver assumes it might be a relative name inside your cluster. So it tries api dot stripe dot com dot your-namespace dot svc dot cluster dot local first. That fails, comes back NXDOMAIN. Then it tries api dot stripe dot com dot svc dot cluster dot local. Fails again. Then api dot stripe dot com dot cluster dot local. Fails again. Only on the fourth attempt does it try api dot stripe dot com as an absolute name, with a trailing dot, and finally succeeds. So a single application-level DNS call just became four queries hitting CoreDNS, three of which were guaranteed to fail. Now multiply that across your fleet. If you're doing a hundred thousand real DNS-triggering requests per second, and a meaningful chunk of those are external hostnames going through this same four-query pattern, your actual query volume against CoreDNS isn't a hundred thousand queries per second. It's closer to three hundred or four hundred thousand, most of it wasted work resolving things that were never going to exist in your cluster domain in the first place. That's the amplification, and it's the single biggest reason CoreDNS falls over under load that looks survivable on paper. There are two levers here, and good candidates know both. First, ndots itself. You can override the search behavior per pod using the dnsConfig field in the pod spec, setting ndots to something lower, like one, for workloads that mostly talk to external services. Or, simpler and often better, you just get your application code or your service mesh sidecar to use fully qualified domain names with a trailing dot for external calls, which bypasses the search list entirely regardless of ndots. That trailing dot is doing real work. It tells the resolver 'this is already absolute, skip the search list,' and it collapses four queries into one. The second lever is the CoreDNS cache plugin, and this is where negative caching becomes the interesting part. CoreDNS caches two kinds of answers: successful responses, and negative responses, meaning NXDOMAIN or NODATA results. By default, the cache plugin's denial caching, the negative cache, often runs with a fairly short TTL, sometimes as low as a handful of seconds depending on your Corefile. That means those three failed lookups per external hostname, the ones hitting cluster dot local variants, don't get cached long enough to actually help you. Every single request re-triggers the same failed chain. The fix is to explicitly tune the cache plugin in your Corefile. You want a reasonably generous success TTL, something like thirty seconds for internal service records that don't change often, and importantly you want to bump the denial TTL too, maybe to a similar range, so that once CoreDNS has established that api dot stripe dot com dot cluster dot local doesn't exist, it doesn't ask again for that window instead of on every single request. The tradeoff you have to talk about in an interview is staleness. If you cache negative responses for sixty seconds and then a service actually does get created with that exact name mid-window, clients will keep getting NXDOMAIN until the cache expires. For genuinely dynamic environments with frequent service creation, you tune this down. For stable production traffic patterns, you can afford to push it up and take the throughput win. This is also where autopath comes into the conversation, because interviewers like asking what it actually buys you versus just tuning ndots. Autopath is a CoreDNS plugin that moves the search path logic from the client side to the server side. Instead of the pod's resolver blindly firing off four sequential queries and waiting for each response before trying the next, autopath has CoreDNS itself walk the search list server-side and return the final correct answer in a single round trip from the client's perspective. From the pod's point of view, it looks like one query went out and one correct answer came back, even though CoreDNS internally still had to check multiple names. The honest tradeoff to mention here: autopath reduces client-observed latency and reduces the number of round trips over the network, but it doesn't eliminate the internal query cost inside CoreDNS, and it adds memory overhead per pod because it has to track pod IP to namespace mappings to know which search path applies. It also depends on the Kubernetes plugin being configured correctly, and it's had stability caveats across CoreDNS versions, so if you cite it in an interview, mention that you'd test it in a staging environment under real load before trusting it in a high-QPS production path. Combined with proper ndots tuning and cache tuning, it's a genuine option, but it's not a silver bullet you deploy blind. Let's talk about the wrong answers people give under pressure, because interviewers are listening for these. The first wrong answer is 'just scale up CoreDNS replicas.' That helps you survive the amplification, but it doesn't fix it. You're paying more compute to serve queries that shouldn't exist in the first place. The second wrong answer is 'disable DNS caching to get fresher answers.' That's backwards. You'd be removing the one thing protecting your upstream from repeated failed lookups. The third wrong answer, and this one sounds smart but isn't, is 'just increase CPU and memory limits on the CoreDNS pods.' That treats a query-volume problem as a resource problem. It buys you headroom, sure, but the moment traffic grows again, you're back in the same spiral, because the root cause, that four-to-one query amplification from ndots, is still sitting there untouched. So here's the thirty-second version. Under Kubernetes defaults, ndots five means any external hostname with fewer than five dots gets tried against your entire internal search domain before it's resolved as absolute, turning one lookup into four. Fix it by using fully qualified names with a trailing dot for external calls, or override ndots in dnsConfig for the workloads that need it. Tune the CoreDNS cache plugin's success and denial TTLs deliberately, don't just accept the defaults, because negative caching is what protects you from repeatedly re-querying names that don't exist. And consider autopath to collapse client-side search path queries into a single server-side lookup, understanding it trades some CoreDNS-side memory and complexity for lower client latency. If you want the full written version of this with the exact Corefile syntax, the dnsConfig YAML, and the follow-up questions interviewers tend to ask after this one, go grab the prep guide at devopsinterview dot cloud.

    CoreDNS at 100K RPS: ndots, Negative Caching, and Autopath

  7. Jul 22

    VPA vs HPA for Stateful Workloads: Autoscaler Deep Dive

    Your Cassandra node is getting evicted at 2 a.m. or your PostgreSQL replica is sitting on 4× the memory it needs — this episode breaks down exactly which autoscaler to reach for and why VPA vs HPA is a staple senior SRE interview question. You'll learn: Why HPA's scale-out model breaks for StatefulSets (token rings, replication slots, unacknowledged messages) and when vertical scaling is the only safe lever VPA's three components — recommender, admission controller, updater — and why update mode auto is the one that pages you at 2 a.m. for stateful workloads The safe progression: start in Off mode for two weeks, apply recommendations manually, then consider Initial mode — and why Auto is rarely worth it for anything with persistent state The HPA + VPA feedback loop failure: both watching CPU on the same workload, pod count oscillating, resource allocation in chaos The clean split that actually works: HPA on queue depth (custom metric), VPA managing memory requests — distinct signals, no interference Keywords: VPA vs HPA Kubernetes, vertical pod autoscaler stateful workloads, autoscaler SRE interview, HPA VPA conflict, Kubernetes StatefulSet autoscaling 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript Your stateful app is getting evicted every few hours, or it's sitting on four times the memory it actually needs, and you're not sure which autoscaler to reach for. That's the exact scenario interviewers love, because most candidates only know one half of the answer. Let's set the stage. You have a Cassandra node, a RabbitMQ broker, or a PostgreSQL read replica running in Kubernetes. It's a StatefulSet. Traffic is not uniform. Some days it's busy, some days it's quiet. Someone on your team says "just add HPA" and someone else says "we need VPA." Who's right? That question shows up in senior SRE and platform engineer interviews constantly, because the answer is not obvious and the wrong choice causes real incidents. Interviewers ask this because autoscaling is one of those topics where surface-level knowledge falls apart fast. Saying "HPA scales pods out, VPA scales pods up" is correct but incomplete. The follow-up is always: okay, so which one do you use for a stateful workload, and why? And then: what happens if you use both at the same time? If you can't answer those, you signal that you've only worked with stateless services. Here's the mental model you need. The HPA, the Horizontal Pod Autoscaler, works by changing the number of pod replicas. It watches a metric, CPU or memory or a custom one, and when that metric crosses a threshold it adds or removes pods. That works beautifully for stateless services. Each replica is identical, sessions don't matter, and spinning up a new pod behind a load balancer is invisible to users. Stateful workloads break that assumption. A Cassandra node owns a subset of the token ring. A RabbitMQ broker may hold unacknowledged messages. A PostgreSQL replica has an open replication slot. Adding a new pod doesn't instantly help because the new pod has to join the cluster, sync data, or acquire state before it can carry load. Scaling out fast is often dangerous. Scaling in is even worse because you might be removing a pod that holds data not yet replicated elsewhere. So for stateful workloads, the more useful lever is usually vertical. Give the existing pod more CPU or more memory so it can handle the load without needing a new replica. That's where the VPA, the Vertical Pod Autoscaler, comes in. VPA has three components. The recommender watches historical resource usage and calculates what your requests and limits should be. The admission controller patches those values into new pods at scheduling time. And the updater, this is the dangerous one, can evict running pods so they restart with the new resource values. The key knob is the update mode, and this is a common interview question on its own. Update mode off means the VPA only calculates recommendations. It never touches your pods. You query the VPA object and apply changes yourself. This is the right starting point for any stateful workload. You get the data without the risk. Update mode initial means the admission controller sets resource values when a pod is first created, but the VPA never evicts a running pod. The pod keeps whatever resources it started with until it's naturally replaced, for example during a rollout. This is a reasonable middle ground for apps that tolerate restarts as part of deployments but not random evictions. Update mode auto, and this is the one that bites teams, allows the VPA updater to evict pods at any time to apply new recommendations. For a stateless deployment this is often fine. For a stateful pod it can be catastrophic. Imagine your primary Redis instance getting evicted at two in the morning because the VPA decided it needed fifty percent more memory. The pod comes back, but during those thirty seconds of restart your application is throwing errors. The fourth mode, recreate, behaves like auto but only triggers on pod creation events. In practice most teams treat auto and recreate as the same risk category for stateful workloads. So the concrete recommendation for stateful apps is: start with update mode off, run it for at least two weeks to cover your traffic patterns, look at the recommended requests in the VPA status, and then apply those values manually to your StatefulSet spec. After you've done that once and confirmed stability, you can consider bumping to initial mode. Staying in auto is rarely worth it for anything with persistent state. Now the tricky part that interviewers really test: can you run HPA and VPA at the same time? The short answer is yes, but only if they're watching different metrics. The classic failure pattern is enabling both on the same workload with both watching CPU. Here's what happens. Load increases. CPU goes up. HPA adds a replica. The CPU per pod drops because the load is now spread. VPA sees lower per-pod CPU and recommends lower requests. It either tells you to reduce requests or, in auto mode, evicts pods to resize them down. Meanwhile HPA is still watching the same CPU signal. You get a feedback loop where both controllers are fighting each other, your pod count oscillates, and your resource allocation is unstable. The safe combination is to give each controller a distinct domain. A common pattern for a message broker is to let HPA scale on queue depth, a custom metric from your metrics pipeline, while VPA manages the memory requests because memory usage on a broker tends to grow with configuration and message size rather than with load spikes. They're watching completely different signals so they don't interfere. Another safe pattern is to disable HPA entirely and rely only on VPA in initial mode, pairing that with a PodDisruptionBudget, the PDB, to prevent the cluster autoscaler from removing nodes that would orphan your pods. This is common for single-instance stateful workloads where horizontal scaling genuinely isn't possible. Let's talk real numbers briefly so you sound credible in the interview. A typical starting point for a moderately loaded RabbitMQ broker might be two hundred and fifty millicores CPU request and five hundred millicores limit, with one gigabyte memory request and two gigabytes limit. After running VPA in off mode for two weeks under normal traffic, you might see the recommender suggesting six hundred millicores CPU and one and a half gigabytes memory. That gap between what you provisioned and what you actually need is exactly the over-provisioning problem VPA is designed to solve. Without it, you're just guessing. On the eviction risk side, the VPA updater respects PodDisruptionBudgets. If your PDB says zero pods unavailable, the updater will not evict your pod even in auto mode. So if you are running VPA auto on a StatefulSet, always pair it with a PDB that reflects your actual availability requirements. This is a detail that separates strong answers from weak ones in interviews. There's also the resource policy inside the VPA spec itself. You can set min allowed and max allowed values per container so the recommender can't suggest something absurd. For example, you can tell it to never recommend less than one hundred millicores or more than four cores, keeping the recommendations inside bounds your infrastructure can actually handle. Always set these bounds for production workloads. One more wrinkle worth knowing. VPA and HPA cannot both target CPU or memory requests at the same time because HPA uses requests as the denominator for its utilization calculation. If VPA changes the request value, the HPA's target threshold shifts underneath it. The percentage-based math breaks. Kubernetes has a known limitation here, and the official guidance is to not use HPA on CPU or memory when VPA is also active on the same workload, unless you're using HPA with custom or external metrics only. Common wrong answers you want to avoid. The first is saying "just use HPA for everything, stateful or not." That ignores the data ownership and sync cost of scaling stateful replicas. The second is saying "VPA is dangerous so never use it." That's overcorrecting. VPA in off or initial mode is genuinely useful and low risk. The third is conflating pod autoscaling with cluster autoscaling. The cluster autoscaler adds nodes. VPA and HPA manage pods within existing nodes. They work at different layers and you need to be clear about which layer you're talking about. The fourth is not mentioning PodDisruptionBudgets when discussing VPA auto mode. Experienced interviewers will notice that gap immediately. Quick recap. For stateful workloads, your default should be VPA in off mode to gather recommendations, and then apply them manually. Use initial mode once you trust the recommendations

    VPA vs HPA for Stateful Workloads: Autoscaler Deep Dive

  8. Jul 15

    Kubernetes Scheduler Extenders: Custom Placement Logic

    Learn how to write a Kubernetes scheduler extender webhook that restricts GPU pods to nodes with NVLink interconnects. This episode covers the extender contract, KubeSchedulerConfiguration registration, filter and prioritize endpoints, and the latency tradeoffs interviewers probe in senior SRE and platform engineering interviews. Full interview prep guides and scenario walkthroughs: DevOpsInterview.Cloud

    Kubernetes Scheduler Extenders: Custom Placement Logic
See All (49)

About

Real DevOps and cloud interview scenarios, broken down the way a senior engineer would answer them. Kubernetes, AWS, Terraform, CI/CD, observability and incident response, with the trade-offs interviewers actually probe for. More prep material at devopsinterview.cloud.

Information