i'm starting to become increasingly skeptical that with all of the free-flow data out there, there is no algorithm that can magically "put it together" in a way that shows you exactly what you want to see.
the problem? what you want to see is mixed in with a whole lot more things you don't want to see, and there is no metadata available that enables anyone to automagically figure out what to show you -- the data's just not there, even for facebook, and if facebook can't do it, twitter doesn't have a chance in hell.
absent of that magic sorting algorithm, "Real Time Data" becomes too time consuming to sort through.
No one said it was easy, and we have a good model of how the industry might evolve: the problem of "search".
We had things like Altavista, Yahoo!, etc, and then came Google with a better algo. It made sense of the data out there without any real use of meta data (meta tags in HTML notwithstanding, at least initially).
And even to this day, we're seeing interesting experiments in solving the problem of finding information on the web, and how to present it. All I'm saying is that the next step is doing this in real time.
i certainly won't exclude that it's in the realm of possibilities... just that, lately, i'm finding it more and more unlikely.
twitter "authority" seems to me to be an even more difficult problem than blog authority, and even that was easily game-able.
besides, twitter doesn't even have the same social graph for me as facebook does, and facebook still has a pretty difficult time separating the wheat from the chaff.
if that breakthrough doesn't show up, it will be revolutionary. if it doesn't, the "real time web" might just be overhyped.
Remember we've only just recently managed to accumulate all this data in a mainstream way. It will take time for that breakthrough to happen, but it will happen.
Example: when AOL released their "anonymous" data, it took the NYT a few days to personally identify a user. It took humans, working on it for a good few hours, being creative, and produced only one data point (one person), but it still happened. We need to find ways to make deep data mining happen much faster... in real time :)
Emphasis on mainstream, by the way, because data mining is not a new field by any measure.
the problem? what you want to see is mixed in with a whole lot more things you don't want to see, and there is no metadata available that enables anyone to automagically figure out what to show you -- the data's just not there, even for facebook, and if facebook can't do it, twitter doesn't have a chance in hell.
absent of that magic sorting algorithm, "Real Time Data" becomes too time consuming to sort through.