> Queues like Kafka can be great for a certain class of solution, but I see them over used.
> The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument.
actually this sort of solution is a great fit to GPGPU, where each thread has some kind of a "window" and will perform N aligned accesses in a stepwise fashion to compute its particular data. Aligned/coalesced accesses and "re-computation" are very cheap in GPGPU and that often means you lean on re-computation rather than notarization or other "stateful" approaches.
I think something like this would be a great fit to AVX-512 probably.
> The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument.
actually this sort of solution is a great fit to GPGPU, where each thread has some kind of a "window" and will perform N aligned accesses in a stepwise fashion to compute its particular data. Aligned/coalesced accesses and "re-computation" are very cheap in GPGPU and that often means you lean on re-computation rather than notarization or other "stateful" approaches.
I think something like this would be a great fit to AVX-512 probably.