In a lot of the comments around this lately, people have been saying that this is something Google can fix, or needs to fix.
I would suggest that the content farms' success in gaming specifically Google's algorithm was an inevitability (whatever the current state of the arms race) and the only thing that will weaken the effectiveness of their techniques is to expose their business model to a greater range of algorithms. If you have three or four search engines all working on slightly different principles, it becomes a lot harder to game them all with the same content, even if gaming any one of them would be trivial. In other words, competition in the SE space at the algorithmic level is something we sorely need to see.
In parallel, my suggestion for one new search engine to add to the mix: a crawler for unsubsidised content. That is, the results consist solely of pages that don't carry advertising of any kind. This wouldn't exclude ecommerce sites but would exclude most kinds of affiliate marketing. Subscriber-only sites could pay to be indexed at a flat rate, though guaranteeing that this fee wouldn't affect rankings might be tricky. Alternatively a journal-access style of subscription model could see the SE paying the content site owner when one of its paying users consumes their information.
The ideal solution would be to make it so that the easiest path for someone trying to abuse Google is simply to be constructive. If your abuse is what people want anyway, then you've won. It's sort of like capitalism: assume that people are nasty sonuvabitches who will lie and cheat their way to fame, money, and power, and then make it so the easiest way to get those is simply to make things that other people want.
Obviously, we're not there yet, and a lot of these spam pages are fairly useless to visitors.
The problem is that it's a tug-of-war situation and in a tug-of-war, you win by passing a minimum threshold; there's no such thing as a runaway victory. Rather than leading to a kind of informational meritocracy, what you're talking about leads to the situation we're in now, where content that is perceived by some proportion of SE users as 'good enough' grows at a rate that is much faster than natural, and drowns out all other content.
If you take advertising out of the equation then the motives for producing general-topic content revert to what they were a few years ago: personal expression, free expert opinion, community discussion, journalism in search of subscribers.
Note that this doesn't require an all-new crawl/engine: just for an existing engine to offer an advanced operator that filters ad-drenched pages from results. Even just an operator that eliminated AdSense sites would be a big win for some queries.
While technically feasible, this would rule out a large number of high quality content sites which are currently ad-supported. For example, specialist community forums often carry ads just to cover costs. You need to allow for a different funding model for sites like that, such as subscriptions.
While I agree that we need to see more algorithmic competition in this space, I fail to understand how any of these new SEs would generate revenue apart from the current ad-supported/PPC model. If these SEs continue to use ads to make money, then the problem of content farm spam would simply migrate to these sites. All they would need to do is to figure out how best to game the new algorithm.
I would suggest that the content farms' success in gaming specifically Google's algorithm was an inevitability (whatever the current state of the arms race) and the only thing that will weaken the effectiveness of their techniques is to expose their business model to a greater range of algorithms. If you have three or four search engines all working on slightly different principles, it becomes a lot harder to game them all with the same content, even if gaming any one of them would be trivial. In other words, competition in the SE space at the algorithmic level is something we sorely need to see.
In parallel, my suggestion for one new search engine to add to the mix: a crawler for unsubsidised content. That is, the results consist solely of pages that don't carry advertising of any kind. This wouldn't exclude ecommerce sites but would exclude most kinds of affiliate marketing. Subscriber-only sites could pay to be indexed at a flat rate, though guaranteeing that this fee wouldn't affect rankings might be tricky. Alternatively a journal-access style of subscription model could see the SE paying the content site owner when one of its paying users consumes their information.