I love this. This is a property already baked into another messaging queuing system called Pulsar and I've thought about writing my own for internal use (maybe to be open-sourced). The only quirk that I see is this link in the project description readme:
> it is guaranteed that each message will be saved in exactly one S3 file
This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.
It's not a 1:1 write per message, it relies on Kafka offset guarantees to batch messages into S3 at a configurable threshold, 256MB per S3 object is common.
Oh! That's awesome. The way that sentence is written is a little tricky to parse, but the behavior of a batching lots of messages (configurable 256MB+ chunks) is exactly what I'm hoping for.
I've wondered for a while now why there isn't something like a cross between Kafka (durable queue broker) and Datomic (a "database" partitioned into storage chunks, where each chunk canonically lives in object storage and is only cached by the instances doing the queries.)
I'd love to see a queuing infrastructure that consists of two parallel clusters, running on the same machines:
• a set of write nodes, which each "own" partitions on a topic (in the sense of being that partition's sole writer); where a write-node receives writes to its partition until it fills up a chunk, at which point it shoves the chunk into object storage;
• a set of read nodes, which serve reads and/or streaming subscription requests by 1. streaming archival data by retrieving chunks from object storage into a read-node-local MRU-on-disk-pressure cache; and then 2. once the read stream "catches up" to the end of the archival data, it would switch to proxying to the relevant write-node, which would serve the query/stream from its uncommitted chunk.
-----
I feel like this queue architecture would be useful for all sorts of things, at least for my own use-cases. Especially because it'd scale perfectly-well down to running one read-node and one write-node on one machine, while still being able to make durability guarantees about the committed data. (The durability of uncommitted data, I don't much need to worry much about, personally.)
Yup, there was definitely a reason to consider Secor before 2017. Part of the reason I'm familiar with Secor in the first place.
Just to be clear, there are dozens of Kafka Connect Sinks and Sources, and finding Kafka Connect Sinks for other blob storage solutions should be easy. If you can't the KC framework makes it pretty easy to develop your own.
At prior company we used Secor to backup Kafka topics to S3 and would use to replay/reingest data with success. If I were to reimplement Kafka somewhere new it would definitely be one of my "Go to Production" requirements.
No. If you want to use Firehose, you need to transfer data from Kafka to Kinesis first. You can probably use one of many Kafka Connect Sinks for this. Eg: https://github.com/awslabs/kinesis-kafka-connector
At that point, might as well set up a Kafka Connect cluster with a Kafka Connect Sink, and dump to S3 yourself.
Not sure if you're being factitious, but it sounded like parent's data was already in Kafka.
But you're right, no reason not to start with kinesis in the first place, especially with all the AWS options around it including Kinesis Analytics (Streaming "SQL"), Flink (supported by AWS now) or plain old EMR (Spark, Flink, etc...).
There is a lot of work that goes into maintaining your own Kafka ecosystem, even if you use Confluent, Lenses, etc...
Good question. Kafka is cheaper for things that don't need to go into redshift, and there's a fair amount of engineering effort that would go in to switching those existing systems over.
We've phased out Kafka for the parts that feed Redshift, since Kinesis Firehose is a far better experience. We still have Kafka for our application logs and performance monitoring
Kinesis Firehose doesn't read from Kafka, we switched our code to publish to Kinesis instead.
> it is guaranteed that each message will be saved in exactly one S3 file
This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.