Aah, the convenience and flexibility of the cloud. I no longer need to worry about expensive hardware purchases, designing my infrastructure to be fault tolerant, capacity planning for peak scenarios, storage, access to AI. The list of benefits goes on, not to mention the constant rollout of new services.
Everything is wonderful, until I need to pull data out of the cloud. Enter the dreaded egress charges. It’s a feeling akin to being held hostage.
It’s my data. Why do I need to pay to get it?
Egress fees seem small from the outset, only pennies. But, not when you’re in the media business and file sizes can be in the 100s of GBs.
There is really no technical reason why it should cost anything to pull my data out of my cloud instances. However, AWS, Google, Azure, IBM, Oracle and most other cloud providers have decided to hold my data hostage.
Why? Because they can. They want to ensure I use their cloud services and one way to do that is to “penalize” me for moving data out.
Don’t get me wrong, I am a capitalist. I understand why it’s done and I believe in market competition. However, I am a capitalist, and I also believe that business owners should do what they can to reduce spending and that means limiting egress costs when possible.
How to Limit Cloud Egress Fees
Some things to consider when evaluating egress fees:
Use S3 compatible storage in your private data center and get some of the Object Storage benefits (simplicity, performance at scale, elasticity…) without moving data to the cloud.
Control your internal developers. Implement policies regarding cloud instances and when they should be used or shut down. I bet when you go on vacation you don’t leave all your lights, appliances and pool heater on.
Implement egress alerts (simple scripts) to send a message when data has reached a certain level
Architect your systems to limit data across cloud regions to minimize costs
Carefully consider which cloud regions to use because the cost of egress can vary
Choose a cloud vendor like Wasabi that does not charge egress
Hybrid-Cloud Use Cases
When looking at egress fees, it’s helpful to consider your overall approach to cloud. In discussions with many companies in the Media Industry, I’ve found that most have taken a hybrid approach to cloud. Their workflows use both the cloud and on-premise resources.
For example, they may have mezzanine content (large files) stored in the company’s private data center in LA that needs to be transferred to London. In addition to the high-resolution video files, a proxy plus descriptive metadata needs to be made available in London.
A common hybrid workflow could look like the following:
A low-resolution proxy is generated on premise in LA and then transferred to public cloud object storage for instance Amazon S3, storing the proxy and metadata.
Cloud services or applications (AI, review and approval etc.) are used to extract additional descriptive metadata from the proxy
From London, metadata is pulled from the cloud incurring very small egress charges (metadata files are typically small)
Mezzanine content is transferred from the private data centers from LA to London.
Mezzanine content is also then transferred to the cloud for an inexpensive Disaster Recovery solution which hopefully the company never needs.
There are many variations to this workflow but they all use the same hybrid approach, and it can be very flexible and powerful.
Various cloud offerings as well as on-premise IT infrastructures have their strengths and weaknesses. Some workflows are ideal for public cloud while others are more efficient and economical if they are run in your private data center.
Developing a Cloud Strategy
When implementing a hybrid approach (or any cloud approach), it’s important to have a strategy and long-term plan that has the flexibility to adapt to changing business needs.
Think carefully about your current architecture, including on-premises and cloud storage as well as the software that moves content between your people and systems. Do your current vendors support your strategy? If not, phase them out with ones that do.
Your plan should make sure that:
Your workflows involving system-to-system automation, cloud regions and people are fully supported
Your file transfer software integrates with all storage types (cloud and on-premises)
You are not locked in to any one cloud vendor
You are able to limit egress fees, as mentioned above
Throughout my career, I’ve been involved with successful partnerships with AWS, Azure and Google, and I am a big proponent of cloud technology and the benefits it brings. But, in order to reap those benefits, there’s a lot that media companies need to consider. Understanding and limiting egress fees is just one aspect, but it’s an important one, especially if it encourages you to take a step back and consider a long-term cloud strategy.