I submitted a bug to Facebook's whitehat disclosure 3 or 4 months ago. Got no response whatsoever, except an automated response. The bug still exists. The bug allows users to post as though they are other users on the timeline. I think that is pretty serious, but I guess they do not.
Great question. It's a long answer and it gets sort of involved.
1) It is easier to get adoption of A/B testing -- which many people have heard of, which many agree that they should be doing, and which captures substantially all of the benefits of bandit testing -- than bandit testing, at the typical company. e.g. If I go to your software company CMO and say "Do you know what A/B testing is?", if the answer is "No", then the CMO is not quite top drawer. "No, and there is no reason why I should know that" is a perfectly acceptable answer for bandit testing.
2) There are some subtleties about actually administering bandit tests, for example in how tests interact which each other or with exogenous trends in your traffic mix, which sound like they could cause operational nightmares. A/B testing does not have 1-to-1 analogues to these problems, and many of the theoretical problems with A/B testing are addressable in practice via e.g. good software and good implementation practices, both of which exist in quantity.
3) A/B testing has vastly better tool support than bandit testing, which currently has one SaaS startup and zero OSS frameworks which I am personally aware of.
4) On a purely selfish note which I'd be remiss in not mentioning, I'm personally identified with A/B testing in a way that I am not with bandit testing.
5) Again, convincing people to start A/B testing will be better 100 out of 100 times than failing to convince people to start bandit testing, which is the default result. Consider the operational superiority of A/B testing for software companies in August 2013, then look at the empirical results: very few companies actually test every week.
(There is also a zeroth answer, which is "I have reviewed the arguments for doing bandit algorithms over A/B testing and frankly don't find them all that credible" but for the purpose of the above answers I assumed that we both agreed bandit was theoretically superior.)
A/B testing is a catch-all term for multi-variant experimentation. Multi-armed bandit is a specific approach to testing[1], and even though most frameworks provide A/B/N testing -- that is, not necessarily just two variants -- it is easier to say 'A/B' instead of 'A/B/N'.
Indeed Google Analytics has added the multi-armed bandit approach in Content Experiment. It's quite slick btw, but definitely more difficult to implement than traditional split testing.
My 2 cents:
> power.prop.test(p1=0.1, p2=0.11, power=0.8,sig.level=0.05)
Two-sample comparison of proportions power calculation
n = 14750.79
p1 = 0.1
p2 = 0.11
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
Edit: so that's 30,000 samples. To do the same with a 5% change in the same area is still 1300 samples (alpha=0.8, p=0.05)
I normally consider relative minimum discernable effect. Your 5% absolute change is a 50% increase is the base rate. I also typically go for a higher power (e.g. 0.9). Under these conditions 60K samples is more typical.
You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power:
> power.prop.test(p1=0.1, p2=0.11, power=0.9,sig.level=0.05)
Two-sample comparison of proportions power calculation
n = 19746.62
p1 = 0.1
p2 = 0.11
sig.level = 0.05
power = 0.9
alternative = two.sided
NOTE: n is number in *each* group
Requires about 40,000 samples per test. I would strongly recommend anyone serious about doing this look in to MAB testing, as A-B testing is way too expensive for reasonable scale testing (unless you have a strong a priori hypothesis to test).
I work with a dozen or so people who are involved in this sort of work. I think it is very interesting to see how they rationalize and deal with their moral compass internally.
One of the guys is the most caring, liberal, loving person you'd ever meet; he justifies being involved in this sort of skeezy marketing work as "I can take a small amount from a lot of people and amplify the result to do good with a lot of money."
He genuinely believes this. A lot of the other guys simply try not to see the "punters" (potential customers [1]) as real people, they are disconnected through the impersonal nature of the internet.
I saw lots of cognitive dissonance when I worked for a company that was something of an advertising firm. Everyone hates pop-ups and spam... but it's probably OK if we do it.
We can turn off the user's pop-up blocker right? What do you mean we can't get around it? How are we supposed to get them to see our side offers if we can't do pop-ups?
And the constant discussions about how to keep our emails from landing in spam folders. Since we weren't running a scam, there had to be a way to get our emails in the inbox right? Even though we don't know the person... bought their address from someone (who bought it from someone)... sent them 3 emails this week already...
I'm glad to not be involved in some of that any more.
Just to show a little balance - there are those of us in advertising who try to do the best we can for our audience, and don't even need morals to make that decision, since doing the right thing gives us better results than trying to trick people.
Gathering data was always a huge problem. The highest traffic site didn't get enough viewes to make any A/B test worthless.
At least in other business likes we could easily track the direction the program was taking and which bits were causing the most problems.
All the end-line employees gated it to various degrees. We
just had to hope we could convince the owners we were rigth.
We did our best to deliver what was requested and not blatantly illegal. When one of your top sales people gets a good deal, you'll find you're doing it to make money, not because anyone thinks it will make things better. Without anyone actually able to figure out what might makes a difference you end up with guesses and snake oil.
With no dada or time to expiremet, all they were left with are terrible ideas like pop-ups and spam. Combine that with the sales guys who talk to buddies elsewhere who report that spam "totally works"! and you end up resenting a large chunk of your job.
That's one of the beautiful things about advertising that a lot of us (well, I assume a lot of us - I can't be the only one, right?) did not realize at first. Especially over the long term, being respectful toward the audience leaves you in a better position to market to that audience. The end result is you make more money.
But there will still be people like me who consider any amount of advertising to be an offense, something that is more desirable if it doesn't exist. You can be as nice about it as you want or feel is appropriate, and I appreciate that, but at the end of the day, your interest is antagonistic to mine.
If you find all advertising offensive then I think that's an issue with you, not with advertising. If you want to look at or use something that doesn't belong to you (whether that's getting on a bus, or visiting a website, or watching a TV channel) then the owners should be allowed to show you advertising if they wish. If you prefer not to see it, you can make that choice and avoid it.
Is it more desirable if it doesn't exist? Perhaps - if you really hate it all then fine, that's your opinion, but do you really hate it enough that you would prefer every website that is financed by advertising to either bill you for usage or shut down, every public transport subsidised by advertising gets more expensive, every TV channel price goes up, cinema tickets, everything...
And as to my interest being antagonistic to yours, I disagree. I (personally) want you to buy a laptop or a PC and I want you to, when you next need one of these items, consider my brand. I target audiences very tightly to try and ensure that people who see my adverts are likely to be interested, and do rather a good job of it if I say so myself. If you end up being interested in what I am advertising (as plenty of people who view them do) then great for me, great for you. If you don't, then you can ignore my adverts, I don't buy any that are overly annoying or evasive, so even without adblock installed they won't cause you any problems, you can just ignore them.
End of the day, my goal isn't to trick you into buying something you don't want, it's to make sure the people who do want to buy these things know about them, and to avoid other people as much as possible. Feel free to hate bad adverts - I do. Even feel free to hate all adverts, including mine, if you wish, though I disagree with you on this. But don't go so far as to call them offensive. They just aren't.
Doing right by the user doesn't mean "doing exactly what the user wants". I'm sure there are people who want iPhones and don't want to pay for them, yet I see no ethical lapse or legitimate conflict of interest with the user in Apple's charging > $0 for an iPhone.
