0.3 Release + Benchmarking Suite

This release refreshes the query engine, swaps in a faster memory allocator, and adds two new ways to get data into a pipeline, along with many other smaller improvements and bug fixes.

This is a breaking change, as plugin interfaces were updated. All plugins will need to be recompiled with the latest streamling-plugin crate.

In our next release, we will be introducing more patterns and operators for complex enrichments, and partitioned processing for better CPU utilization.

Benchmarks

We created a benchmarking suite to catch regressions and compare Streamling against Bento and Flink on the same workloads.

The clearest way to see the difference is throughput per unit of memory. Streamling led with roughly 40x Flink on the Kafka workload and about 3x on the heavier Parquet → ClickHouse workload. On raw throughput, Streamling matched or surpassed Flink on Kafka and surpassed Bento by 2-5x across the board, while Flink took the lead on single-replica ClickHouse writes.

Streamling was built for stable and efficient performance on pipelines that power high-stakes financial use-cases, and in persistent production settings, stable and efficient resource usage is a key anchor for uptime. This benchmark will help make sure Streamling doesn't regress.

Kafka + SQL Projection

This benchmarks a read from Kafka (Avro via Schema Registry) → select every payload column plus lower(country) and one arithmetic column → discard data. It's focused on decoding and multi-transform data passing over sink writes.

Streamling leads on both throughput and efficiency, and the gap widens as machines get smaller:

Flink had different pod configurations as we wanted to make use of the parallelism settings to reflect real-world usage, which required 4 GB minimum. It was unable to run on the lowest machine setting.

Bento was the most memory-light, but was CPU-bound through the tests, resulting in lower throughput.

Streamling consistently had the lowest CPU usage and best memory-to-throughput ratio, resulting in the highest resource efficiency generally. However, Streamling has room to improve. The next release will focus on efficient data partitioning in order to have many internal 'copies' of the pipeline, allowing us to fully saturate resources.

Parquet -> ClickHouse benchmark

This benchmarks a much heavier workload: field-boundary GeoParquet (~218 GiB / 3.2B rows across 1000 files) read from a public S3 bucket → rename and flatten the nested fields, hex-encode the WKB geometry, derive an md5-based dedup key → ClickHouse Cloud ReplacingMergeTree.

This is a test against single deployment efficiency. The Streamling numbers use parallel processing.

Flink wins on absolute throughput by saturating more of the provisioned CPU as parallelism is dialed up. Streamling uses substantially less CPU and memory for the same workload, but leaves headroom unused in a single replica.

However, in real usage, Streamling users spin up multiple replicas when higher throughput is needed. In production we've observed that this scales linearly, and given the resource headroom in these results, we expect multi-replica Streamling can surpass Flink's single-deployment throughput. We did not include a multi-replica mode in this suite, since the goal for the next release is to bring those efficiencies inside one deployment.

Bento could not be tested at comparable resource limits. The Parquet path materializes each whole S3 object as a single in-memory batch, so the densest file alone needs 30+ GB live. It required a 110 GB limit on a dedicated node (70 GB observed peak) to finish at all, while the other engines stream the same data inside 13 GB. This test will be re-run when the streaming parquet source is available.

Full results are in the benchmarking suite.

What's new in 0.3

DataFusion v54

Streamling now builds on the latest Apache DataFusion (v54), picking up its query-planning and execution improvements across the board.

Significant Kafka Source improvements

Avro decoding moved onto Arrow's native Avro support with avro-arrow. Records now decode straight into Arrow record batches instead of being parsed row by row and reassembled.

In production at Goldsky, pipelines reading from Kafka Avro topics saw an immediate 20% improvement with less CPU being used and higher throughput for the same workload and pipeline size.

Memory usage efficiency

After doing extensive benchmarking on mimalloc v2, jemalloc, and mimalloc v3, we found that mimalloc v3 lowers allocation latency and improves multi-threaded throughput, which translates to steadier performance under load and much lower memory usage.

Combined with an audit on memory usage and zero-copy enforcement across the codebase, we saw significant memory reductions for all running pipelines.

In Goldsky's cloud, which runs thousands of Streamling pipelines, an example customer saw a memory efficiency improvement from 31 GB to 4 GB on a workload that writes large (up to 1m rows) batches in real-time into ClickHouse.

New sources: File and Kafka JSON

Two new ways to read data:

• File source — point a pipeline at local or object-store files (Parquet, JSON, CSV, or Avro). Run it as a bounded source for backfills and replays, or as a continuous source that keeps watching the path for new data.
• Kafka JSON — the Kafka source now decodes JSON payloads in addition to Avro, so you can consume JSON topics without a schema registry.

Both look like any other source in your pipeline YAML:

sources:
  # Backfill from newline-delimited JSON files.
  seed.events:
    type: file
    path: /data/events
    format: json
    primary_key: id
    mode:
      type: bounded

  # Consume a JSON Kafka topic — no schema registry required.
  live.events:
    type: kafka
    topic: events.clicks
    data_format: json
    primary_key: id
    schema:
      id: string
      url: string
      user_id: string

ClickHouse Sink Optimization

Streamling's native ClickHouse sink sends Arrow format to ClickHouse, allowing for huge savings by skipping serialization. ZSTD and LZ4 are now added as compression codecs on top of Arrow, with ZSTD yielding significant efficiency improvements for most scenarios.

For large batches (100k rows per batch), sink flush times went down by 50%.

Previously, GZIP was used, but after testing with customer workloads the team found that ZSTD provided the best performance generally.

For small batches, sink flush latency is slightly higher. Combined with the network egress savings users will likely see, ZSTD was chosen as the default, but LZ4 and GZIP remain as options.

If you value the fastest writes over bandwidth savings, for example in low-latency or co-located deployments where you aren't network-bound, you can disable compression or select LZ4. In practice the gains are usually marginal.

Postgres dynamic tables

Dynamic tables are a way to efficiently filter against set of values kept in a database. In production, users use dynamic tables with millions of addresses. Dynamic table lookups against Postgres got two improvements that tend to land together in enrichment-heavy pipelines.

Before, for each batch, a lightweight query against Postgres was done. It would contain a deduplicated set of values from streamling, get the results, and filter the incoming batch. An opt-in cache can keep the full set of in process so repeated keys don't round-trip to Postgres on every row, while keeping the same consistency. A cursor check is used instead of a primary key update.

Separately, dynamic_table_check now accepts text[] input with any-match semantics, useful when a row carries several candidate keys and you want a hit if any of them resolve. This allows for efficient filtering of nested values.

Byte-size batch flushes

The batch accumulator can now flush on a byte-size threshold, not only row count or time. That matters for wide or variable-size rows, where a fixed row limit either underfills the sink or blows past memory before the timer fires. This is very useful large backfills, where disk throughput needs to be managed.

Variable-size records are an important target to handle in use-cases with a lot of text or patterns where computations are stored in arrays inside of one row. Allowing for high throughput on denormalized data will allow architects to directly operationalize raw data instead of going through a multi-step pre-cleaning process.

This is a good default setting to use for all sinks as well, and will make it easier to get to the optimal throughput.

Also fixed

• Postgres sinks recover from read-only failover (SQLSTATE 25006)
• Checkpoint markers are preserved through multi-sink transform chains
• Postgres statement execution is bounded client-side, so stuck statements don't hang the pipeline
• Avro writer supports Arrow LargeUtf8
• Short jobs flush the delta meter provider on shutdown
• Arrow take offset overflows are handled instead of failing hard
• Script-transform schema validation is panic-free and accepts uint / bytes aliases
• Kafka JSON schema errors stay user-facing under --validate
• Empty metric metadata and unknown plugins return validation errors instead of panicking
• Unregistered preprocessor ids warn and skip instead of failing startup

Also in this release: the file source can ingest a single file and leverage partitions, and sink primary key deduplication can be turned off when you don't need it.

Upgrade

Install or upgrade with the one-line bootstrap:

curl -fsSL https://streamling.dev/install.sh | bash

The full changelog is on GitHub.