The call out on premature optimization is valid. However this article misses the mark on a couple fronts.
One, as others have called out, is the ability to control rollout (and rollback) without needing a deployment. Think mobile apps and the rollout friction. If something goes wrong, you need a way to turn off the offending functionality quickly without having to go through another deployment or a mobile app review.
Second, is to be able to understand the impact of the rollout. Feature flags can easily measure how the rollout of one feature can affect the rest of the system - whether it is usage, crash rates, engagement, or further down the funnel, revenue. It’s a cheat code for quickly turning every rollout into an experiment. And you don’t need a large sample size for catching some of these.
By having this power, you will find yourself doing more of it, which I believe is good.
Feature flags are not 100% reliable. They tend to be booleans of two different values. We've had complications on the null path, e.g. we assumed all mobile use would be either from Android or iOS, but sometimes an Android device refuses to identify as Android.
The fallback has to be handled properly. We make sure the default is the one with least risk and have to audit this regularly to fit the current state of the product. Yes, they're tech debt, there's also management overhead for this, and they should be regularly cleaned up. There was a habit of using it as a hack for unprepared releases, so we put some bureaucracy to make sure it's properly thought out.
They're also a security risk; hackers can just send a false flag, so we have to consider whether this is safe.
A lot of the use of feature flags is because a customer asked for a certain feature. For these, it's usually better to just hardcode it to if (customer===A)
This is a good list that includes a lot of things most people miss. I would also suggest:
1. Tight targeting of your users in an AB test. This can be through proper exposure logging, or aiming at users down-funnel if you’re actually running a down-funnel experiment. If your new iOS and Android feature is going to be launched separately, then separate the experiments.
2. Making sure your experiment runs in 7-day increments. Averaging out weekly seasonality can be important in reducing variance but also ensures your results accurately predict the effect of a full rollout.
Everything mentioned in this article, including stratified sampling and CUPED are available, out-of-the-box on Statsig. Disclaimer: I’m the founder, and this response was shared by our DS Lead.
> 2. Making sure your experiment runs in 7-day increments. Averaging out weekly seasonality can be important in reducing variance but also ensures your results accurately predict the effect of a full rollout.
There are of course many seasonalities: day/nigh, weekly, monthly, yearly seasonality, so it can be difficult to decide how broad you want to collect data. But I remember interviewing at a very large online retailer and they did their a/b tests in an hour because they "would collect enough data points to be statistical significant" and that never sat right with me.
Hey all, creator of Small Basic here. I made this back in 2008 as a side project when I used to work at Microsoft. My belief was/is that you can get more kids interested in programming when the IDE (language + runtime + libraries) encourages iterative coding with simple gratification loops built in.
You can see a bunch of decisions influenced by .Net which I was working with back then. I had been meaning to go back and revisit some of those decisions, but haven't had a chance yet. I have changed my stance around scoping and argument passing in functions, for instance.
(What is really neat is that 15 years later, my son is learning coding with Small Basic. And I hired an engineer for my current startup who had started coding with Small Basic as a kid in Brazil.)
In the 1980s, home computers like the Sinclair Spectrum and Commodore 64 booted straight into a programming environment, often BASIC. Even if you just wanted to play games, you got some mandatory exposure to coding.
I think it would be great if there was a modern "console computer" like that, which let you code the instant you switched it on.
At the very least, I wish modern programming environments let you use graphics and draw right away without fumbling around with boilerplate, like a simple 10 LINE(1,1,10,10) :)
Yep. C64 basic was my gateway into programming. I typed in almost all of my games from Compute and Compute's Gazette. That was until I joined a C64 user group and started getting pirated stuff.
Nice to hear from you, great work! As a lifelong dev who started by learning BASICA and who's first pro job was in Visual Basic, I love your thesis.
Nowadays I work with people living with disabilities and I come across kids who want to program but who cannot use a traditional keyboard and mouse. I was just discussing the need for a simplified ide and language syntax with full programming power and Small Basic came to mind.
Any chance this work will see new life outside of its 'side project' status?
I took SB to the next level in Small visual Basic. Its source code is published, and it consists of 7 WPF (VB .NET) projects, so you can fork the language and modify the form designer and code editor as you like:
https://github.com/VBAndCs/sVB-Small-Visual-Basic
As with anything, there was a long list of work that I never got around to mostly because life happened - kids, startups, etc. I would love to bring Small Basic to the browser where it's easy to share your programs and creations with your friends.
I continue to maintain the webservices needed for the app, but any substantial investment is at least a year out unfortunately.
Fast growing Series B startup, hitting strong traction with customers. Looking for generalists that can go up and down stack, and for Infra engineers that can build scalable systems - we process 7B+ events today, and doubling every few months.
One, as others have called out, is the ability to control rollout (and rollback) without needing a deployment. Think mobile apps and the rollout friction. If something goes wrong, you need a way to turn off the offending functionality quickly without having to go through another deployment or a mobile app review.
Second, is to be able to understand the impact of the rollout. Feature flags can easily measure how the rollout of one feature can affect the rest of the system - whether it is usage, crash rates, engagement, or further down the funnel, revenue. It’s a cheat code for quickly turning every rollout into an experiment. And you don’t need a large sample size for catching some of these.
By having this power, you will find yourself doing more of it, which I believe is good.