The Situation
Kubernetes makes it easy to run workloads at scale. It also makes it easy to spend more than you need to without noticing until the bill arrives.
For one of our clients, a SaaS company running dozens of services across multiple EKS clusters on AWS, that bill had been climbing for months. Engineering teams were growing, new services were being deployed regularly, and infrastructure costs were scaling faster than the business itself. Nobody could explain exactly why, and nobody fully owned the problem.
They brought us in expecting a capacity conversation. What we found was a visibility and standards problem.
The Problem
The clusters were running. Services were stable. But underneath that stability was a significant amount of waste baked into every layer of the infrastructure.
Resource requests and limits had been set once, during initial deployment, and never revisited. Teams had learned early on that requesting more CPU and memory meant fewer OOMKills and throttling incidents, so the default behaviour over time became to overprovision. It felt safer. Nobody was penalised for it. And because there was no cost visibility at the team or namespace level, there was no feedback loop to encourage better decisions.
The symptoms were clear once we started looking:
- Average CPU utilisation across clusters was sitting around 20%, meaning the nodes were paying for capacity that was almost never used
- Memory requests were routinely set at two to three times actual consumption
- Node autoscaling was configured but poorly tuned, scaling up quickly and scaling down slowly, leaving underutilized nodes running for hours longer than necessary
- Spot instances were not being used anywhere, despite a large portion of workloads being well suited for them
- No team had meaningful visibility into what their services were actually costing, making it impossible to hold anyone accountable or make informed decisions
The result was a cluster that was technically healthy but financially inefficient. Every new service added to the problem because teams copied existing resource configurations without questioning them.
Our Approach
We approached this in layers, working from the node level down to individual workloads, and closing the loop with proper cost visibility.
Phase 1: Rethinking Node Provisioning with Karpenter (Weeks 1–3)
The clusters were running the standard Kubernetes Cluster Autoscaler. It works, but it has well-known limitations. It scales based on pending pods, is slow to consolidate and deprovision underutilized nodes, and has limited awareness of the cost implications of the nodes it provisions.
We replaced it with Karpenter, which provisions nodes directly in response to pod scheduling requirements rather than relying on predefined node groups. This shift had immediate effects:
- Karpenter selected the right instance type for each workload automatically, matching node size to actual demand rather than defaulting to a fixed instance family
- Node deprovisioning became far more aggressive, with Karpenter continuously consolidating workloads and terminating underutilized nodes rather than waiting for thresholds to be breached
- We configured Spot instance support through Karpenter's NodePool policies, routing non-critical workloads and batch jobs onto Spot nodes with automatic fallback to on-demand capacity when needed
- Node provisioning time dropped significantly, reducing the window where pods sat pending during traffic spikes
Within the first few weeks, node count during off-peak hours had dropped noticeably and the instance mix had shifted toward smaller, better-matched node types that reflected actual workload needs rather than assumptions made at initial setup.
Phase 2: Right-Sizing Workloads with Goldilocks and VPA (Weeks 2–5)
Fixing the nodes only solves half the problem. If pods are still requesting far more resources than they actually use, Karpenter will simply provision correctly-sized nodes to host incorrectly-sized pods.
We introduced Goldilocks, a tool built on top of the Kubernetes Vertical Pod Autoscaler, to generate data-driven resource recommendations for every workload. Goldilocks runs VPA in recommendation mode alongside each deployment, observes actual CPU and memory consumption over time, and surfaces the results in a clear dashboard that makes it straightforward to see where requests are misaligned with reality.
The findings were consistent across almost every service. Memory requests were set at roughly two to three times the recommended value. CPU requests were more variable but still consistently higher than actual usage patterns justified. We used the Goldilocks recommendations to:
- Right-size resource requests and limits across all production workloads in a controlled, staged rollout
- Establish new internal standards for how teams should set resource requests when deploying new services, turning one-off fixes into lasting defaults
- Enable VPA in auto mode for selected lower-risk workloads where continuous right-sizing made sense without requiring manual intervention each time
This phase required careful change management. Teams were understandably cautious about reducing resource allocations on services they were responsible for. We ran all changes in staging first, monitored closely, and shared results transparently so teams built confidence in the process before anything changed in production.
Phase 3: Improving Scaling Behaviour with HPA and KEDA (Weeks 4–6)
Several services had no autoscaling configured at all, running at a fixed replica count sized for peak traffic regardless of actual load. Others had Horizontal Pod Autoscaler configured but tuned to CPU thresholds alone, which did not reflect how those services actually experienced demand.
We reviewed and reconfigured scaling behaviour across the service fleet:
- For CPU-bound services, we tuned HPA target utilisation thresholds to scale down more aggressively during quiet periods without risking performance during traffic spikes
- For services where CPU was not a meaningful proxy for load, we introduced KEDA to enable event-driven autoscaling based on application-level signals such as request queue depth, message lag, and latency percentiles sourced from Prometheus
- For services with predictable traffic patterns, including batch jobs and internal tooling, we implemented scheduled scaling to proactively reduce replicas during known low-traffic windows rather than waiting for HPA to catch up
The combination of better scaling behaviour and right-sized resource requests meant the cluster was now provisioning and reacting based on reality rather than worst-case assumptions that had been baked in years earlier.
Phase 4: Cost Visibility That Actually Worked (Weeks 5–7)
None of the changes above would be sustainable without giving teams the ability to see what their services were costing. Without that feedback loop, resource configurations would drift back over time as new services were added and old habits returned.
We implemented cost visibility using two complementary approaches.
The first was OpenCost deployed alongside the existing Prometheus and Grafana stack, providing real-time cost allocation broken down by namespace, deployment, and label. Teams could now see what their services were spending, how that compared to previous periods, and how their resource requests related to actual usage without needing to leave their existing observability tooling.
The second was enabling AWS Split Cost Allocation Data for EKS through AWS Cost and Usage Reports. This is a native AWS capability that distributes EC2 instance costs at the Kubernetes pod level based on actual CPU and memory consumption, using tags such as cluster name, namespace, workload type, and deployment name to make costs attributable across the organization. Unlike third-party tooling alone, this approach integrates directly with AWS billing data, meaning finance and engineering teams were looking at the same numbers from the same source of truth.
Together these two layers gave the team both the operational granularity they needed day to day and the financial accuracy required for meaningful cost attribution and chargeback conversations across teams and business units.
The change in behaviour was visible almost immediately. Teams that had never considered resource efficiency started treating their resource requests as something they owned and were accountable for, rather than a configuration detail set once and forgotten.
The Results
Within twelve weeks, the infrastructure looked fundamentally different in terms of cost and efficiency, without a single performance regression or customer-facing incident caused by the changes.
- Cloud infrastructure spend reduced by 40% compared to the three-month average before the engagement
- CPU utilisation across the cluster increased from around 20% to over 55%, reflecting workloads running on appropriately-sized nodes
- Spot instances now accounted for around 45% of total node capacity, concentrated on workloads where interruption risk was acceptable
- Deployment-related resource incidents, OOMKills, and throttling events dropped by over 70% as resource requests became more accurate rather than just large
- Every team now had a live cost dashboard for their namespace, and onboarding a new service included resource configuration guidance as a default step rather than an afterthought
Beyond the numbers, the platform team stopped being the only people who thought about infrastructure cost. That shift in ownership is what makes the results durable.
What Made It Work
We fixed the node layer and the workload layer together. Addressing only one side of the problem would have produced partial results. Right-sized pods on oversized nodes still waste money. Efficient pods on a poorly-tuned autoscaler still leave idle capacity running.
We used data to change behaviour, not just configuration. Goldilocks and the cost dashboards gave teams something to look at and act on. The technical changes were necessary but the visibility was what made the improvements stick.
We tied engineering tooling to financial reporting. Connecting OpenCost to AWS Split Cost Allocation Data meant the same cost story was visible to engineering leads and finance teams alike. That alignment removed a lot of friction from conversations about cloud spend.
Where CosmosGrid Fits
At CosmosGrid, we help organizations reduce cloud costs without sacrificing reliability or engineering velocity. This includes Kubernetes cost optimisation, platform engineering, autoscaling strategy, and infrastructure standardization.
Our approach focuses on finding the waste that has built up quietly over time and replacing it with systems that stay efficient as the organization grows, not just on the day the engagement ends.
Final Thoughts
Kubernetes cost problems rarely come from a single bad decision. They accumulate gradually as teams make safe, locally reasonable choices that add up to a globally inefficient system. Nobody intended to overprovision. Nobody chose to run on oversized nodes. It just became the path of least resistance.
The organizations that address this will not only reduce their cloud bills. They will build infrastructure that scales more predictably, gives teams clearer ownership, and creates the kind of visibility that makes future decisions easier to get right. Cost efficiency and performance are not in conflict. In most cases, the same changes that reduce waste also improve reliability. The cluster that runs lean is usually the one that is best understood.
