> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.
I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.
Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?
Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.
That or they just didn't dimension well their prices and now it's biting them back.
I prefer to assume naivety over malice/negligence.
Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering.
The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".
As an engineer, if you don't qualify whether your answer is average or max, you've messed up.
I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.
As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.
I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.
> I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan.
I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".
Then why does this only apply in the US? Are they saying EU customers are well behaved?
It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.
- in EU there are large peering exchanges to swap traffic
- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.
Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.
Hetzner colo's for their US servers/locations. Perhaps their US bandwidth rates are much higher than their own DCs?
probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.
Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".
“Culprits” because they used the service they paid for within its advertised limits?
It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?
They are simply deflecting blame for their own enshittification.
>“Culprits” because they used the service they paid for within its advertised limits?
"Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.
So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.
Sure .. also them.
Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.
It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
It’s not any more possible to correctly anticipate all pricing structure vulnerabilities, than it is to correctly anticipate all program and API security vulnerabilities. There is always a statistical chance of novel outcomes when humans are involved.
8 years ago, hetzner had no cloud offerings
The dedicated servers still have 20 tb traffic included
Want to give a clue on who the culprit is?
I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.