Petabyte-Scale S3 Optimization with 70% Cost Savings

At Opslyft, we work closely with growing SaaS businesses that rely heavily on AWS storage. In this case, the customer operated a global video hosting platform for enterprise sales and marketing teams. With nearly one million full HD videos, their storage footprint approached 10 petabytes.

Although Amazon S3 provided the scalability and reliability they needed, costs were rising steadily. Our goal was simple: understand the architecture, identify cost drivers, and redesign the system so that the storage strategy matched real access behavior. What followed was a structured optimization process that reduced their Amazon S3 bill by approximately 70 percent.

Understanding the Architecture and the Hidden Cost Risk

The platform used a Just-In-Time Processing model. Instead of storing multiple versions of each video in different formats, the system generated the required format only when a viewer requested it. This reduced storage duplication and initially appeared cost-efficient.

To further lower storage costs, the team configured a lifecycle policy in Amazon S3. After 30 days, older videos were automatically moved to S3 Glacier Instant Retrieval, a storage class designed for data that is rarely accessed but must remain available within milliseconds. On paper, this made financial sense.

For example, storing 1 terabyte of data in Glacier Instant Retrieval costs roughly 4 dollars per month, compared to about 21 dollars in S3 Standard. However, Glacier Instant Retrieval charges additional fees every time an object is retrieved. If that same 1 terabyte is accessed even once, retrieval and GET request costs can total around 30 dollars.

The assumption was that older videos would rarely be watched. In reality, some files continued to receive heavy traffic. As a result, retrieval charges erased the expected savings.

This led us to two key questions:

• Were too many frequently accessed videos stored in Glacier Instant Retrieval?
• If so, how could we identify them and correct the issue efficiently?

For a detailed technical breakdown, you can read the original AWS case study here.

Using Data to Identify the Real Cost Drivers

We enabled Amazon S3 Access Logging to capture detailed records of every request made to the storage buckets. These logs were then analyzed using Amazon Athena, allowing us to query access frequency at scale.

Within hours, the data revealed a clear pattern. A very small percentage of video files, roughly 0.1 percent, accounted for nearly half of all GET and retrieval activity in Glacier Instant Retrieval. These were typically larger marketing videos in high resolution, such as 1080p or 4K.

Although they represented only a small fraction of the total 10 million objects, they generated a disproportionate number of requests. In fact, around 10 percent of objects were responsible for approximately 99 percent of the 3.1 billion monthly GET requests.

Storing these high-traffic files in Glacier Instant Retrieval was inefficient. We evaluated moving them to S3 Intelligent-Tiering, which does not charge retrieval fees and has significantly lower GET request costs. Cost modeling showed potential savings of over 90 percent for the most accessed files.

We flagged the top 60,000 active objects, about 0.6 percent of total content, and moved them to S3 Intelligent-Tiering. This change alone reduced retrieval and GET costs by roughly 50 percent.

Optimizing Storage Strategy with Multiple S3 Classes

The deeper insight was that not all videos behave the same way. Some content remains rarely viewed, while others attract sustained traffic over time. A single storage strategy could not serve both patterns efficiently.

We restructured the model as follows:

• High-traffic, large marketing videos were stored directly in S3 Intelligent-Tiering at upload.
• Lower-traffic content remained in S3 Standard for the first 30 days.
• After 30 days of low activity, files transitioned automatically to Glacier Instant Retrieval.

This required only minor code adjustments but produced a significant financial impact. GET request volume gradually shifted away from Glacier Instant Retrieval, and overall S3 costs began to decline steadily.

Reducing GET Requests Across the Pipeline

After optimizing storage classes, we focused on another major cost driver: the sheer number of S3 GET requests. Even with proper storage placement, excessive request volume increases expenses.

We examined two layers of the system:

1. Content delivery layer

The platform used Amazon CloudFront as its content delivery network. Analysis of Amazon CloudWatch metrics showed a global cache hit rate of only 65 percent, with some regions dropping to 40 percent.

A low cache hit rate means more requests fall back to the origin, which in this case was Amazon S3. After tuning the CloudFront distribution settings and improving regional configurations, the cache hit rate increased to approximately 90 percent.

This improvement alone reduced S3 GET and retrieval requests by about 50 percent.

2. Packaging layer optimization

The Just-In-Time packaging layer, based on Nginx, regenerated video segments whenever CloudFront experienced a cache miss. Because it did not cache files locally, each segment required multiple S3 GET range requests.

Originally, each segment triggered an average of 7.05 GET requests. By increasing the byte range size from 256 KB to 2 MB, we reduced that average to 1.04 GET requests per segment.

This change reduced GET request volume by approximately 85 percent at the packaging layer.

Combined with storage optimization and CDN tuning, total S3 GET requests dropped by around 90 percent.

Final Outcome and Key Takeaways

Through detailed cost analysis using AWS Cost and Usage Reports and S3 Access Logging, combined with architectural adjustments across storage and delivery layers, the platform reduced its six-figure annual Amazon S3 bill by roughly 70 percent.

The most important lesson from this engagement is clear. Cost optimization on AWS is rarely about removing services. It is about aligning architecture with real usage patterns.

In large-scale environments, small percentages matter. A fraction of objects can drive the majority of the cost. Without detailed visibility, those cost drivers remain hidden.

At Opslyft, we approach optimization through architecture review, data-driven analysis, and targeted adjustments. When storage classes, access patterns, and content delivery layers are aligned correctly, organizations can scale confidently while maintaining financial control.