That doesn't quite sound like what the content of the article says - it's more, "We used the two algorithms from the first Progress Prize that gave us most of the benefit, and the 107 blended algorithms required to get the next 1.7% improvement weren't worth it. Oh, and here's how we had to reengineer them." (Numbers from memory, not to scale.)
The title makes it sound like the prize ended up being pointless. The article says otherwise.
I work on the cinematch team at netflix. The several hundred blended algorithms that was output of the grand prize winning team is very... impractical... for netflix' needs. We cherry picked (and then modified) the best of the bunch.
Netflix's blog post mentions they used two algorithms from one of the Progress Prize ($50,000) winners.
The article goes on to say "...you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later...We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment."
Accurate, but a half truth. The prize was for a 10% improvement, but before that solution was produced they had already improved by 8.4%. The headline makes it sound like the improvement from zero to 10% was not worth the engineering cost, but really it was the improvement from 8.4% to 10% which cost too much.
In the Netflix prize, did all teams merge into one group and stop trying? Nor would we expect so with K groups, especially with exponentially decreasing prize amounts. If this is even slightly a risk, you can compensate by making the maximum number of prizes awarded a function of the number of participating teams. For example, you can award 10 prizes if there are >100 teams, and if not award floor(num_teams/10) prizes, with the prize pool redistributed among the top 10%.
It was pure luck that the threshold for the million-dollar prize was crossed. If it had been arbitrarily set at 11% (as opposed to 10%), then there's a good chance the million dollar prize would have never been paid out.
The advantage of paying out K top prizes is that other teams that don't win outright may have developed additional useful models or insights (or used more computationally efficient algorithms), and you may access to these in this manner.
Yes, the engineering costs are too hard to justify... now that DVDs by mail is no longer their primary focus. it's not clear to me that that would still be the case if Netflix business were still operating around 2007's constraints.
Did the entry ranked 5 also have the 2 algorithms? Then the title makes sense. A contest is a good way to motivate the base, but it can also cause lots of effort that is unnecessary in the end.
The title makes it sound like the prize ended up being pointless. The article says otherwise.