cullenking 6 days ago

I didn't downvote you FYI, but to answer your question, I run ridewithgps and as a result have a pretty in depth understanding of all the costs involved.

We have 14 machines in a rack at a datacenter in PDX, and have focused on low hosting costs since we have historically been bootstrapped and margin sensitive. Redundant switches, 2 firewall/load balancers, 4 compute machines, 3 database machines, 5 storage servers. Single upstream network provider, about to be two sometime next year. Rack space + power + redundant network is about $3500 a month. Machines have an average service life of about 5 years.

Database machines cost about $45k for a set of three.

Storing user data is non-trivial - GPS track files add up when you get close to a billion of them, photos are also very large. We use a self-hosted ceph object storage cluster of 5 machines, about $100k of hardware. it's cheaper than 20k a month in S3 bills.

Our rack all-in is probably about $250k of equipment. 5 year service life, probably $5k a month amortized out. So, doing things as cheap as possible (I buy nvme SSDs for storage cluster off ebay, and am about to buy a couple arista 100gbe switches from ebay as well) we are somewhere around $8500 a month on hosting.

We use both google maps as well as self-hosted OSM based map and routing services. About half our map usage goes to google, by user preference, and we pay them about $20,000 per month for that. Our self-hosted OSM map stuff require 1tb of ram, fast disks, a ton of CPU cores. We host 10 different planet scale routing profiles via graphhopper, which take 3 days to build every week with updated data. They also host a vector maps stack which is much more efficient, taking about 3 hours a week to build.

My last estimate of an AWS bill for all the above was $30k a month, assuming some discounts. We have grown since then and I'd napkin us to be > $40k a month at this point.

We strive to minimize any costs for third party platforms. We do use amplitude for analytics, that's > $30k a year at this point. We do use an external email service for easy marketing emails, but the majority of our millions of emails a month are sent from our own mail servers, using an in-house system we made a decade ago that still works well. We try to minimize vendor lockin and costs, where it makes sense.

Most expensive part of the entire company of course is salaries, with of course an eye to developer and related salaries. We run pretty bare bones where possible, with a flat management structure with minimal overhead. Our total staff size is 32, of which 6 are full-time end user support.

A bit of a ramble, sorry, but there's a large amount of overhead to run a system like ours. We don't just make a one-time use desktop application, we have to continually provide storage and compute for all users. If we stopped that, the entire service would fall apart. So yes, a subscription makes sense in a case like ours. You can't do what we do with a desktop app. Plus, the entire world has switched to mobile for this sort of consumer application, which is an entire rat race of it's own. You can't just release a single purchase app and expect it to be maintained, it's a massive effort to keep up with mobile development just to maintain features, much less build anything new.

4
wintermutestwin 6 days ago

I sincerely appreciate your serious answer to my serious question. I am quite surprised at all these costs considering that OSM is free, but you explained them very thoroughly and I am impressed. You have adequately justified the subscription requirement.

This post, combined with the fact that your planner actually allows me to force a path has won me as a customer. I'll also note that setting your yearly at 75% of the monthly is wise considering very low winter time usage.

For me, churn or not is going to come down to whether I can read critical details on your mobile app without having to put reading glasses on. (which is a factor that not even a $2t company like Apple can address properly)

cullenking 6 days ago

Unfortunately we still have work to do there. We have dabbled in a separate map style, but it's really difficult to get any roadnames to show up when you start increasing font sizes in map styles. It's a sort of pick-your-poison - big maps where you can see the roads with almost no road names, or maps with smaller details with road names. The app itself should mostly scale well with increased system font sizes, but we are still carving out a couple of webviews in the app which have funky behavior. I do like to joke that with our average staff age passing 40 in the next year or two, we'll solve all readability issues....

wintermutestwin 6 days ago

Well, at least you are aware and considering oldster eyes. Apple regularly makes incredibly bad UI choices that would be easy to avoid.

Serious cyclists tend to be on the older side…

dx-800 6 days ago

Thanks for your interesting comment. I'm a happy customer of ridewithgps, and run a small B2B SaaS, so I love hearing details like this.

One comment: I hope you don't start emphasizing the "social media" aspects, like Strava has done, to their detriment, in my opinion. That's what prompted me to finally delete my Strava account (after uploading my thousands of activities to ridewithgps.)

kemotep 6 days ago

Always great to see a successful “Single Rack of Failure” project out in the wild. And I don’t mean this as a negative. By being able to contain and control all the business into a single rack, you can more easily set up redundancy than trying to replicate your AWS environment between availability zones or worst, try recreating it in Azure or Google with all the different services, footguns, and so on. Just get another rack in a different datacenter and you’re set.

Congratulations to your team on keeping costs low and running a successful business out of a single rack.

cullenking 6 days ago

We are just about finished with a total hoist into a self-hosted k8s cluster. We've gone slow due to my concerns about k8s and the additional complexity, but it's actually been pretty smooth and enjoyable. The end goal is the ability to point our helm charts at any cloud provider and have the entire infra stood up in less than a day. We already do offsite backups to rsync.net. We don't need crazy redundancy and are OK with a certain amount of risk to availability and < 1 day of data loss. Only thing that would take a long time is some GIS search data. We use elasticsearch and h3 hex strings encoded to guarantee prefix searching works for different zoom levels, which is a couple terabytes of derived data that takes a while to compute.

I have been enjoying this on-prem renaissance that we've seen over the last couple of years, makes my stubbornness around self-hosting feel smart in hindsight ;)

Moru 6 days ago

You forget one thing. RWGPS is (I think) one of the few training apps that can record and display all the data on the free tier. And it doesn't nag you every 5 minutes to upgrade to Pro or whatever. I really appreciate that and try to buy a one-month subscription once per year when I can afford it.

I have also worked with gps data and run my own map server (for a small country) so I know it costs money, time and effort. And thanks for the breakdown!