I have worked at two startups now where we made the fatal mistake of being profitable. If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure, because we're scaling up to 5 billion users of course.
Of course we started spending all the money on new people and AWS, and soon there was no money.
At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS.
I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.
AWS or not, investors invest for the sole purpose of spending money with the hope of finding a unicorn. If an investor just wants to sit on their money, they can do that without giving it to a company.
Then those investors are stupid. Hope is not a strategy. But more likely is this take is wrong. Also sounds to me like GP maybe missed another consideration: architecting your application in such a manner that scaling is straightforward (not easy, straightforward, there is a difference).
Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console.
Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.
On the other side of the spectrum, AWS's extensive cost report metrics via tagging are great for big companies.
I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.
...sure, as long as they tag their resources properly.
The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes.
The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes.
ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.
AWS cost reporting is far from great.
Its hard to learn current(daily) charges; RI is completely hidden and only visible in the final bill, blended; there's no way to limit the spending and the detailed reports are in csv, not user-readable.
Yes, there aren't great AWS provided tools, but the data is there. We happen to use Cloudability, though I'm sure there are other good tools, maybe even free ones.
A startup being able to spend 400k/month on AWS should also be able to employ somebody to keep an eye on the AWS cost and look for possible optimizations. If that's not the case I wouldn't blame AWS for the spent money.
No blame was on AWS. The TAMs tried to help fix things but no one was listening. Many times I was the only person meeting the 2-3 TAMs we had on the account.
I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically
Not sure what the monthly revenue was but I do know there were several multimillion dollar deals every 6 months or so.
Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know.
No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance.
I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.
If you're taking money from investors and you're interested in being profitable, then you're going to sacrifice growth for the sake of profitability:
- Let's not hire for this idea because it eats into our burn
- Hmm, let's hold off on launching this feature until we have more data
Investment $ means taking risks (within reason) to maximize shareholder value.
If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution.
It sounds like the founders at your company were just inexperienced.
Dude. A bunch of billionaires just got away with completely tanking the economy and getting bailed out for it. Zero people ever even talked about the possibility of anyone going to jail. Everyone got their bonuses. Nobody suffered any consequences. Literally nobody.
If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.
That would require amazon gathering customer information and possibly GDPR data on the usage of all of it's customers, which is a thing amazon doesn't do, and you don't want your cloud providers doing.
Of course we started spending all the money on new people and AWS, and soon there was no money.
At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS.
I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.