Show HN: DuckDB for Kafka Stream Processing
by dm03514 on 12/8/2025, 5:25:54 PM
Hello Everyone! We built SQLFlow as a lightweight stream processing engine.<p>We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory!<p>DuckDB also supports a rich ecosystem of sinks and connectors!<p><a href="https://sql-flow.com/docs/category/tutorials/" rel="nofollow">https://sql-flow.com/docs/category/tutorials/</a><p><a href="https://github.com/turbolytics/sql-flow" rel="nofollow">https://github.com/turbolytics/sql-flow</a><p>We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors<p>I would love your feedback, criticisms and/or experiences!<p>Thank you
https://sql-flow.com/docs/tutorials/intro/
Comments
by: srameshc
This looks brilliant, thank you. I love DuckDB and use it for lot of local data processing jobs. We have a data stream, not to the size where we need to push to BigQuery or elsewhere. I was thinking of trying something like sql-flow but I am glad now it makes the job very easy.
12/8/2025, 6:24:37 PM
by: mihevc
How does this compare to <a href="https://github.com/Query-farm/tributary" rel="nofollow">https://github.com/Query-farm/tributary</a> ?
12/8/2025, 6:38:41 PM
by: mbay
I see an example with what looks like a lookup-type join against a Postgres DB. Are stream/stream joins supported, though?<p>The DLQ and Prometheus integration out of the box are nice.
12/8/2025, 6:32:29 PM