A long standing complaint of mine is that Cloud egress pricing severely limits the usefulness of compute. If I want to say process some visual effects on a large (1TB) ProRes video, I might spend $1 on the compute but $100 on the egress getting it back.
Unfortunately these changes don't really resolve that problem. "Standard" pricing is a paltry 20% less. That 1TB video egress still costs $80 and for that price I can rent a beefy server with a dedicated gigabit pipe for a month.
Why is "Cloud" bandwidth so damned expensive?
I'd love a "best effort" or "off peak" tier. I imagine Google's pipes are pretty empty when NA is asleep and my batch jobs aren't really going to care.
I was going to back up my photos to AWS Glacier before noticing that retrieval costs are multiple times the storage cost. I guess that is possibly OK for a backup but scared me into a physical alternative.
AWS has greatly simplified their Glacier pricing from the inscrutable transfer rate pricing to more common sense latency based pricing.
If you want your data back in < 5 minutes it's $0.03/GB, 3 - 5 hours, $0.01/GB, 5 - 12 hours, $0.0025/GB
AWS Glacier is cheaper per month than Google ColdStorage, but Google doesn't charge more for faster access to your data. So maybe you still need that Calculus to figure out which is a better deal for your data access pattern.
Also nice to have sub-millisecond latency to first byte (versus multi-hour) and performance characteristics of normal GCS.. so you could run a rare MapReduce job on it just the same as regular GCS :)
99% availability seems not so "highly available" as they call it. It's just SLA, but that means if the service is down for 7 hours every month it still meets the availability SLA.
I know it may not matter that much for backup, I'm just wondering why would they set the bar so low - i.e. how do they organize the storage that they had to set it so low. I'm guessing MAIDs but I still don't get it.
You just said that you were looking to Glacier as a "back up", so isn't that a good use case for Glacier?
One advantage of Amazon versus many other cloud backup providers is that (for a fee), they'll ship you your data on a hard disk if you need it in a hurry. Having cheap access to your 4TB of data may not be such a great deal if it takes 3 months to download it.
I'm fairly certain that the calculator you linked is incorrect. Unless I'm not understanding the Glacier pricing page, it should cost at most $1,240 to store 10TB for 30 days and then re-retrieve over the internet (even at the expedited retrieval rate).
The calculator shows the same setup (albeit with some of the rate information hidden) as costing $3,934.91
Backblaze B2 has no retrieval cost, so the same setup (storing 10TB for a month, then downloading) would cost only $250 (less than 1/3 of the cheapest Glacier option). Glacier is really only a viable option if you're sticking within the same region (where transfer is free) or you plan to literally never retrieve your backups.
Otherwise, B2 and other alternatives are much cheaper.
I'm trying to use the nomenclature from the Glacier pricing page, where "transfer" is the verb to reference the cost per GB downloading the data from the remote provider. Glacier has an additional "retrieval" cost per GB that B2 and others do not (because they most likely have to physically retrieve a tape/HDD drive with your data on it).
Sorry I don't get it. With B2 I have to pay if I want to say download 100GB of data in B2. From what I read, you say this isn't so. But the pricing page state otherwise
As a suggestion, I use Glacier (with Arq) as a tertiary backup. If I have to go to that it means a lot of other things failed and I'll be willing to pay the price.
FWIW Amazon drive will store any number of photos (or anything it thinks is a photo -- I havent looked to see if it looks at the file name or checks to see if it's a valid jpeg/png) for free. So if you're really backing up your photos that's an option.
Yeah, I organized my backup around it. Then they decided that maybe they won't offer unlimited storage after all, and I better pay up or they are going to delete my files. Not falling for that again.
I just checked the terms of service and they don't at all. They did ban rclone a while back but they said it was because of poor security in rclone itself, and haven't banned anything else.
Interestingly they say you can't use the storage for commercial purposes (whatever that means).
rclone was sharing an API key publicly. Not really a security flaw but understandable to block. Except that they shut down the ability for anyone to get API keys a while ago, so users can't get their own.
I wonder what that same 1 TB of data costs on the "open market" (or how it is even priced).
Say I'm a smallish ISP on the east coast and want to send 1 TB to another smallish ISP on the west coast. I probably have a contract with, I don't know, AT&T or whoever owns the cables (I'm from Europe so I have no clue, in Germany it would be Telekom). I'm guessing it would probably cost cents (and that would bring my data across the continent, not just out Google's door)? Definitely nowhere near $80.
In the open market you either get transit (you buy bandwidth from a carrier) or peer (you exchange traffic with a provider) with other networks.
There are also IXs, where you can peer with a lot of different networks in one place (DE-CIX is the biggest for example, located in Frankfurt).
Transit is billed 95th percentile, that means if you want to commit to 1 GbE on a 10 GbE uplink you pay the 1 GbE outright and can burst to the 10 GbE the uplink provides you. If you are over 1 GbE at the end of the month (95th) you will be billed the remaining mbps with a per mbps pricing.
Now, to answer your question, it basically works like that:
You are present in NYC and you want to reach a customer on the west coast that uses Comcast. You would buy transit in your NYC facility from Comcast to get a direct path/way to them. They will take care of everything else. You don't have to lease/rent/buy the dark fibre (cables) from the east to the west coast, the provider (in this case comcast) is doing that already. If you don't buy transit from comcast, but instead form some cheap carrier (say cogent) they will do the same BUT not guarantee a specific bandwidth to comcast, you only would get 10 GbE to cogent, but you don't know if you can push the 10 GbE to comcast.
Transit providers also vary vastly on price. DTAG (Telekom) for example is ridiculously expensive (5x the prices of regular transit providers).
Let's say you want an decent pipe (transit) with 1 GbE commit on 10 GbE, you would pay something like $500-$600 for the 1 GbE commit and after that around $0,50 per every additional mbps. Additionally you need a router, preferably two (redundancy). Fiber cables, transceivers, etc. All those things are very expensive.
What you also notice is that you can't get transit with a TB billing, but most providers offer you per GB/TB billing. They need to commit to a certain amount and hope they calculated well in regards to their commitments and usage etc etc. Anyways, $80-120 per TB, the cloud providers are charging, is a bad joke.
With enough competition a smallish ISP would pay as low as $0.25-$0.40 per mbps per month, which is on an order of $1 per TB. More expensive offers are like $1-$2 per mbps per month, which is still a single digit per TB.
Intentionally not linking to any of the companies I found with just a quick search at google (mislaid the PDF for carrier 2 carrier pricing of a major provider): 40E/month for 30TB traffic, 1G uplink (1G/s guaranteed), 2x500G SSD on a i7.
This is one of the easiest ways for cloud providers to make money. I very often see people doing lots of advanced calculations for compute and choosing providers based on that without really thinking about bandwidth usage.
Famous last words, "Oh, it's a web service so bandwidth shouldn't be a big deal."
They could simply ban adult websites in their ToS, I doubt this is the reason. Also seed boxes are nowhere near useful if you still would pay $10 / TB and at the same time can get traffic for free at OVH or Online/Scaleway.
It's most likely vendor lock-in and people are dumb enough to pay for it.
The reason, is that bandwidth is a solid profit center for the cloud majors. Amazon initiated it, the rest are going along in cartel fashion, nobody wants to disrupt one of the lucrative angles by driving that profit center toward zero.
There's no other rational explanation why Azure or Google aren't hammering AWS on bandwidth cost as a competitive angle. They're basically conspiring in the industrial sense, no different than airlines that somehow magically agree to simultaneously raise prices or department stores that for decades all agreed not to discount the perfume etc. box section of their stores.
No need for conspiracy theories. When one market participant anchors a price at a high premium, you don't need to collude to know that undercutting them is a scorched earth strategy that causes both of you to lose. Part of this is because financial markets value your revenue more than market share, and undercutting bandwidth pricing hurts the former more than it helps the latter.
High bandwidth costs also make it more costly to move some of the data processing to companies which offer very cheap dedicated servers.
--
But it's not fair to say that price from Google is high just because some random hosting company gives you unmetered 500mbs for $99/month. As we have seen with many unlimited cloud storage offerings, some companies do unsustainable offers to attract customers. There are different strategies behind this. Some just count on that majority of customers won't use everything they could. Others know the price hike is inevitable, but believe customers will still stuck.
Save your outputs in the cloud. There is zero cost to access S3 object storage from EC2. Its only if you insist on pulling down 1TB files to your local workstation that you run into this. This seems like a self-inflicted problem.
The problem is often I need that data locally. With video editing in particular, I can't work with data that's not local and I can't run my editing software in the cloud because I'm not aware of any kind of remote workstation that offers colour management and whatnot.
I think part of the reason is people look at Linode/Vultr/DO and presume that 1/2TB for $5 is the market price (and then quote HE.Net/Cogent/Hibernia offering 10G to certain colos at $2-5kpm as backing proof), or dedicated servers with 10TB for $50pm etc... Linode/Vultr/DO mainly work off an average usage of 1%-2%, although at least they dont stop you deploying more boxes as you reach the limit, you can get clever and one would presume they dont invoke some TOS clause. I imagine GCP, AWS, and Azure bandwidth cost is closer to the truth at running a network that is resilent and can scale beyond 100G, the transit or costs of fiber is probably only a tiny fraction when pushing beyond 100G per region, vs your low end VPS provider, let's not forget the epic fail of Linode DDOS handling. I do agree though in comparison it sounds expensive.
No, I can assure you that Linode, OVH, DO are making money per G at 1TB per $5.
Cloud providers are expensive because they want vendor lock in. They want to make it artificially cheaper to use their own services (elasticsearch, etc) over something else. And they want a high switchover cost when you want to transfer all of your data to another provider.
Unfortunately these changes don't really resolve that problem. "Standard" pricing is a paltry 20% less. That 1TB video egress still costs $80 and for that price I can rent a beefy server with a dedicated gigabit pipe for a month.
Why is "Cloud" bandwidth so damned expensive?
I'd love a "best effort" or "off peak" tier. I imagine Google's pipes are pretty empty when NA is asleep and my batch jobs aren't really going to care.