I've tried to setup this up at a small tech company pre-kubernetes and there were a lot of challenges...
* low utilization, so you need to auto-shutdown or auto-scale down systems but sometimes jobs did need to run overnight or on the weekend so you needed an interface/user training to avoid upsetting users/killing their jobs.
* local hardware is cheap and powerful - employees are already issued really powerful laptops and some teams just went out and bought their own really, really powerful workstations. It is hard for the 'cloud' to compete with this.
* bin-packing workloads was hard, Kubernetes probably solves this better.
* fast-customization was hard, we had docker but it was hard to train users to update/fork the dockerfiles to keep
the environment reproducible. A lot of users were more scientists than engineers and so weren't great at using version control.
* persistence / shared-storage IOPs are expensive - shared storage is a nice to have and a number of teams made a lot of use of it, it also made it easy to migrate users around but it's expensive. Local disks were also very painful/slow/buggy to deattach/reattach (maybe this is better now).
* latency - we needed instances close to our team and at the time the metro clusters near us were like second class with low capacity and limited features.
* specialty hardware like GPUs
* multiple paradigms, we also needed long running dev/staging environments, spark clusters, or other software, some managed or licensed to run in specific way, and it was hard to get these all managed the same way, clustered on the same nodes without introducing other issues - but w/out this costs would spiral. Again now that kubernetes is like the defacto cluster manager this might be easier now.
Yeah I can't really get beyond this thing about how powerful our machines on our desk are compared to what hosting providers charge if you have 12 months utilization (I get people in SV change jobs every 6 months but).
I guess the thing that makes this make sense for Uber is the ginormous repo? This probably is a good "big ball of mud" solution to a bazillion tiny projects, each with varying degrees of documentation making it impossible to run.
One day we as an industry will figure out how to make it so that our dev setups work and run well in a multitude of environment (hopefully without that solution involving "we are pinning to a specific Ubuntu docker image"...)
> Yeah I can't really get beyond this thing about how powerful our machines on our desk are compared to what hosting providers charge if you have 12 months utilization
I'm not so sure. Here's a quick comparison:
Desktop:
Dell XPS
12ᵗʰ Gen Intel® Core™ i5-12400
Windows 11 Home
Intel® UHD Graphics 730
8 GB, DDR5
256 GB SSD
Suppose I work every working day of the year for 10 hours (an overestimate). I would pay 250 * 10 * $0.06 = $150 for the Linode. So the Dell desktop takes 4.5 years to pay for itself.
Granted this is a fairly rough comparison but I don't think it's obvious that physical machines are more cost efficient that cloud machines.
First off these specs are... poor. My 2012 spare laptop that I keep for nostalgia has more memory and storage.
You can't physically connect a local screen and a keyboard to a remote Linode instance, so the $0.06/h is on top of the $669.99 (which in your example doesn't include screen, input, etc).
So in that exact setup you end up with a sub-par local computer that connects to a sub-par remote computer (and creates all kinds of headaches associated with remotely accessing compute resources); where the remote computer can't do a single thing that a ($669.99+$150) $820 local computer wouldn't do much better.
All to save you how much cash? How many hours' worth of $ in engineer salary? $2000-4000 will get you a really powerful M1/M2 Mac, and if you need more than 128GB of ram then you can indeed spin up an instance - and instances with >128GB of memory are nowhere near close to $0.06/h.
Optimize for developer productivity, that is your bottleneck. Uber's problem is less of "our local machines are bad and/or expensive", more of "our setup is so convoluted, and we have so many devs, that we need to apply economies of scale to tackle the problem".
> $2000-4000 will get you a really powerful M1/M2 Mac
If we take this as the basis for comparison, then the same dollars will buy 1-2 (working hours only) years of Linode 96 GB with 20 cores, so it seems like cloud services indeed lose competitiveness at the high end.
Not a great comparison. The i5-12400 has 6 physical P-cores. The Linode instance you mention has 4 vCPUs, which typically correspond to logical cores and are subject to noisy neighbors. If your workload needs those CPUs, a better comparison would be against the “Dedicated 32 GB” plan, which has 16 dedicated vCPUs. Those instances are 3x your quoted price.
> If your workload needs those CPUs, a better comparison would be against the “Dedicated 32 GB” plan
I'm not sure what you mean. The "Dedicated 32 GB" plan is 6x my quoted price. Why wouldn't the comparison be to the midpoint of the "Dedicated 8 GB" and "Dedicated 16 GB" plans (which have 4 and 8 CPUs respectively)? That would be around $0.14 per hour, about 2.5x my quoted price.
Anyway, it seems highly unlikely that local development, which uses CPU very burstily, needs a dedicated plan.
When your app is really 50 microservices, or one really big monolith, there's no great way to run it locally. And if you have one single job that takes up more RAM than you have, or requires tons of parallel CPU, the laptop is a bottleneck. And heaven forbid your corporate overlords saddle you with horrible virus-scanning crap or mandatory HTTP proxies with custom certs...
A lot of this will be solved if the people making our OSes ever decide to build a real distributed OS (either SSI or something more like P9). The OS is supposed to make programs easier. Why we keep trying to reinvent the wheel inside every application, and tie them together with duct tape, I don't know.
* low utilization, so you need to auto-shutdown or auto-scale down systems but sometimes jobs did need to run overnight or on the weekend so you needed an interface/user training to avoid upsetting users/killing their jobs.
* local hardware is cheap and powerful - employees are already issued really powerful laptops and some teams just went out and bought their own really, really powerful workstations. It is hard for the 'cloud' to compete with this.
* bin-packing workloads was hard, Kubernetes probably solves this better.
* fast-customization was hard, we had docker but it was hard to train users to update/fork the dockerfiles to keep the environment reproducible. A lot of users were more scientists than engineers and so weren't great at using version control.
* persistence / shared-storage IOPs are expensive - shared storage is a nice to have and a number of teams made a lot of use of it, it also made it easy to migrate users around but it's expensive. Local disks were also very painful/slow/buggy to deattach/reattach (maybe this is better now).
* latency - we needed instances close to our team and at the time the metro clusters near us were like second class with low capacity and limited features.
* specialty hardware like GPUs
* multiple paradigms, we also needed long running dev/staging environments, spark clusters, or other software, some managed or licensed to run in specific way, and it was hard to get these all managed the same way, clustered on the same nodes without introducing other issues - but w/out this costs would spiral. Again now that kubernetes is like the defacto cluster manager this might be easier now.