There is an argument for Plasma/State channels here though, where transaction speed and fees become an issue only when users cash in and out of a state channel? It works for some business models, not necessarily all. I just feel like the Ethereum community is aware of this, and moving in the right direction.

I own https://www.DomainDetect.io, we pull new domains from Zone files/3rd party sources daily. ~200,000-400,000 domains per day to alert on new phishing sites for businesses. So we do something similar. I would start with something like https://czds.icann.org/en attempting to resolve them asyncronously in your country, then work your way through the new daily additions. Use an ElasticSearch/Cassandra backend for speed to query that volume of information.

Your problem will come from the countries/TLD's that dont play ball with their zone files. These are trickier to obtain.