A walkthrough of EKS control plane logging for security investigations: which of the five log types actually capture who did what, how to write CloudWatch Logs Insights queries against audit logs, and how to build a metric filter alert on sensitive verbs like delete and create against secrets, roles, and the aws-auth configmap. Visit https://devopsinterview.cloud for interview prep ebooks, roadmaps and hands-on incident labs. Transcript A compliance audit flags a handful of API calls on your EKS cluster that nobody can explain. Someone created a cluster role binding last month. Someone else deleted a secret three weeks ago. The security team wants a name, a timestamp, and a source IP, and you go looking for the trail only to find that nobody turned on the logging that would have captured any of it. That gap is exactly what this question is testing, and it comes up constantly in SRE and platform interviews because it sits at the intersection of Kubernetes internals, AWS billing tradeoffs, and actual incident response. Interviewers ask this because almost everyone can say the phrase enable control plane logging, but very few can explain which of the five log types actually gives you an audit trail versus which ones are just operational noise. It also tests whether you understand that CloudTrail and Kubernetes audit logs are answering two completely different questions, which is a mistake that trips up even experienced engineers. Here's the mental model. EKS gives you five control plane log types: api server, audit, authenticator, controller manager, and scheduler. The controller manager and scheduler logs are mostly useful for debugging cluster behavior, like why a deployment isn't scaling or why pods are stuck pending. They're not what you want for a security investigation. The api server log shows you raw requests hitting the API server, which is useful but incomplete on its own. The two log types that actually matter for a compliance breach are audit and authenticator. The audit log is the Kubernetes API server's own audit trail. Every request that mutates or reads cluster state gets recorded with the verb, like get, create, update, patch, or delete, the object reference, meaning the resource type and namespace, the user identity, the source IP, the user agent, and a timestamp. This is your who-did-what record. The authenticator log is specific to EKS and shows you how IAM identities get mapped into Kubernetes RBAC through the aws-auth configmap or access entries. If someone's IAM role suddenly started acting like cluster-admin, the authenticator log is where you see that mapping happen. A critical point that trips people up in interviews: CloudTrail does not give you this. CloudTrail logs calls to the EKS control plane itself, things like CreateCluster, UpdateNodegroupConfig, or DescribeCluster. It has no visibility into what happens once a request reaches the Kubernetes API server inside the cluster. If someone uses kubectl to delete a secret or create a role binding, that's a Kubernetes API call, not an AWS API call, and CloudTrail simply won't see it. This distinction alone is a good signal in an interview because it shows you actually understand where the AWS control plane ends and the Kubernetes control plane begins. Now the walkthrough. First you enable the logging. Using the AWS command line, you'd run something like aws eks update-cluster-config, specifying your cluster name and a logging block that turns on the api, audit, and authenticator types, and explicitly turns off controller manager and scheduler if you don't need them. Once that's live, EKS starts streaming logs into a CloudWatch log group named something like slash aws slash eks slash your-cluster-name slash cluster, with separate log streams per type. Here's where the cost conversation matters, because interviewers love asking about this tradeoff. A moderately busy cluster, say fifty nodes with a healthy mix of workloads and controllers reconciling state, can easily generate two to five gigabytes of audit log data per day. CloudWatch charges roughly fifty cents per gigabyte ingested, plus storage costs on top, and audit logs are chatty by nature because every single API interaction, including read-only list and watch calls from system components, gets recorded. If you don't set a retention policy on that log group, you're paying to store that indefinitely. A sensible default is something like ninety days retention for audit logs to satisfy most compliance frameworks like SOC 2 or PCI DSS, then export older data to S3 if you need longer retention for legal reasons. Once the logs are flowing, you go to CloudWatch Logs Insights to actually investigate. This is a query language, not full text search, and it's worth knowing the syntax cold for an interview. A basic investigative query looks like this: you select fields for timestamp, the username under user, the verb, the object reference resource, the object reference namespace, and source IPs. Then you filter where verb is in the list delete, create, update, and where the object reference resource equals secrets. Then you sort by timestamp descending and limit the results. That single query answers the exact question compliance asked: who touched secrets, when, and from where. You'd run a similar query filtering on object reference resource equals clusterrolebindings or rolebindings to catch privilege escalation attempts, because creating a binding to the cluster-admin role is one of the most dangerous single actions someone can take in a cluster. You'd also want a query that filters specifically for changes to the aws-auth configmap, since that's the bridge between IAM and RBAC, and any unexpected edit there deserves immediate attention regardless of who made it. Investigation is reactive though, and the real value in an interview answer is showing you'd build proactive alerting instead of waiting for an auditor to ask questions. The pattern here is a CloudWatch metric filter on the audit log group that matches your sensitive verb and resource combination, for example delete against secrets, or create against clusterrolebindings, or any write verb against the aws-auth configmap specifically. That metric filter increments a custom metric whenever a match occurs. You attach a CloudWatch alarm to that metric with a threshold of one event over a five-minute window, and the alarm publishes to an SNS topic that fans out to Slack or PagerDuty. For teams doing more sophisticated response, you route the same metric filter through EventBridge into a Lambda function that automatically pulls the full audit record, checks the source IP against your known CIDR ranges, and opens a ticket automatically if it's an unrecognized source. Let's talk about the common wrong answers, because these show up constantly. The first is enabling only the api server log and assuming that's sufficient, when it actually gives you less structured identity information than the dedicated audit log. The second is treating CloudTrail as the audit trail for in-cluster activity, which as we covered, it simply isn't. The third is enabling everything, all five log types, without setting retention or filtering, which quietly turns into a massive CloudWatch bill within a month with no one noticing until finance asks questions. The fourth mistake is writing alerts that fire on every single write operation, which drowns your team in noise from legitimate CI/CD pipelines doing routine deployments, and within two weeks everyone starts ignoring the alert channel entirely. Good alerting here means scoping tightly to specific verb and resource combinations that are rare and high-risk, not broad categories. One more nuance worth mentioning if the interviewer pushes further: audit log entries include a stage field showing whether the record is the request received, the response started, or the response complete. For investigations you generally want the response complete stage, because that tells you whether the mutating action actually succeeded, versus a request that was received but denied by RBAC. Filtering only on request received would falsely flag denied attempts as successful actions, which matters a lot when you're trying to figure out whether an actual breach occurred or someone just got a permission denied error. So to recap. EKS gives you five control plane log types, but audit and authenticator are the ones that answer compliance and security questions, while controller manager and scheduler are for operational debugging. CloudTrail covers the AWS-level EKS API, not Kubernetes API calls inside the cluster, so don't confuse the two in an interview answer. Enable logging through cluster config, set a sane retention period to control cost, and use CloudWatch Logs Insights queries filtering on verb, object reference resource, and username to trace exactly who did what. Then move from reactive investigation to proactive alerting with metric filters on sensitive verb and resource combinations, especially around secrets, role bindings, and the aws-auth configmap, routed through SNS or EventBridge so you catch the next incident in minutes instead of during next quarter's audit. If you want to drill this kind of scenario until the answer comes out clean under pressure, head to the DevOps Interview Cloud channel on YouTube for daily thirty-second interview drills, and check devopsinterview dot cloud for the full write-up on this one.
