
CLIENT OVERVIEW
CHALLENGES

High Inter-AZ Costs
The client’s multi-zone EKS clusters were incurring high costs due to data transfer between Availability Zones (AZs).

Expensive S3 Transfers
S3 data transfers were occurring over the public internet, leading to elevated network costs.

Kafka Management Costs
Running Confluent Kafka led to large network expenses.

Inefficient Use of Spot Instances
EKS was running stateless workloads with minimal spot instance usage.

S3 Bucket Policies and Data Retention
Lack of bucket policies and ineffective data retention led to unnecessary storage costs.

High Network Charges with Vendor
Extensive data transfer costs due to using a vendor without efficient data access methods.
Solution Implemented
OPTIMIZED EKS CLUSTERS
Problem:
High inter-AZ data transfer costs from multi-zone EKS clusters.
Solution:
We reconfigured the setup to have one EKS cluster per AZ. We also ensured AWS ELB (Elastic Load Balancer) served traffic from only one AZ, significantly reducing cross-AZ data transfer costs.


S3 ENDPOINT IMPLEMENTATION
Problem:
S3 data transfers over the internet led to high networking bills.
Solution:
We set up S3 endpoints within the client’s VPC, enabling data transfers to occur within the AWS network and reducing internet egress costs.
I N-HOUSE KAFKA MANAGEMENT
Problem:
High network costs associated with managed Confluent Kafka.
Solution:
Client’s Confluent cluster was setup to talk over the internet. There was no easy way to move this cluster from public to private which would have allowed us to setup direct connection with Confluent. Client was interested in bringing Kafka in-house. We migrated Kafka management in-house using AWS MSK (Managed Streaming for Apache Kafka). We optimized topic retention policies and broker configurations to reduce operational costs. Cost of Confluent cluster including storage and network was $700-800k, move to MSK saved ~$50k/month.

SPOT INSTANCES FOR EKS
Problem:
Low utilization of cost-effective spot instances for stateless EKS workloads.
Solution:
We collaborated with our partners to implement spot instances where feasible, cutting down on compute costs.
RESERVED INSTANCE PRICING WITHOUT COMMITMENT
Problem:
The client had a substantial non-spot EC2 footprint and was looking for ways to secure the best possible EC2 rates.
Solution:
We collaborated with one of our partners who specializes in providing EC2 instances at Reserved Instance (RI) discounted rates without requiring any long-term commitments. By leveraging this partnership, we helped the client save approximately 45% or $240k/month in EC2 costs.


ELASTICACHE REDIS COST OPTIMIZATION
Problem:
The client was facing high costs with their ElastiCache Redis setup.
Solution:
We collaborated with the in-house team to review and adjust the Time to Live (TTL) settings on objects stored in Redis. By reducing TTLs, we freed up memory, which enabled a reduction in the instance size. This optimization led to significant cost savings.
PRIVATE LINK SETUP
Problem:
High network charges from frequent data access via a third-party vendor.
Solution:
We established a PrivateLink connection with the vendor, which reduced the network traffic costs by facilitating private, secure data access.
S3 DATA MANAGEMENT
Problem:
Absence of S3 bucket policies and excessive data retention.
Solution:
We reviewed and adjusted S3 bucket policies, setting appropriate object retention times. This helped in cleaning up unnecessary data and controlling storage costs.
ONGOING COST MANAGEMENT
TERRAFORM MODULES FOR BEST PRACTICES
Problem:
Inconsistent resource provisioning led to potential inefficiencies
Solution:
We implemented AWS Config rules to generate compliance alerts. These alerts ensure that any infrastructure changes align with predefined standards and best practices, providing an additional layer of oversight and helping maintain a cost-effective and compliant environment.
COMPLIANCE ALERTS WITH AWS CONFIG
Problem:
There was a lack of automated compliance monitoring for infrastructure standards.
Solution:
These alerts ensure that any infrastructure changes align with predefined standards and best practices, providing an additional layer of oversight and helping maintain a cost-effective and compliant environment.