PipeKeelBeta

Streaming

Stream Kafka into BigQuery in real time

PipeKeel reads Kafka topics (Confluent Cloud, Amazon MSK, Aiven or self-hosted, with SASL or client certificates) and writes them to BigQuery, databases, APIs or other topics. In real time a stream worker writes every few seconds; in micro-batches every block, including joins and aggregates, is available. Offsets are committed only once every target has the records, so a failure re-reads instead of losing data.

Start free See pricing

How it works

  1. Connect the cluster

    Bootstrap servers plus SASL credentials or a client certificate and your provider's CA certificate.

  2. Describe the records

    List the JSON fields to read (with types), or take each record as text; add partition, offset, timestamp and key if you need them.

  3. Add steps and targets

    Filter, derive, cast, mask personal data, flatten JSON — then BigQuery or any other target.

  4. Choose real time or micro-batches

    Real time for seconds of latency; micro-batches every few minutes for joins and aggregates. Pause and resume any time.

What you get

  • Real time or micro-batches, chosen per pipeline
  • At least once: offsets move only after every target has the rows
  • Sovereign: the stream worker runs in your project, reaching private clusters
  • SASL/PLAIN, SCRAM and mutual TLS (Aiven's default)
  • Status on the pipeline: rows written, lag, last batch

Questions

How do I stream Kafka to BigQuery without writing code?

Connect the Kafka cluster in PipeKeel, add the Kafka source and a BigQuery target to a pipeline, choose Delivery → Real time, and activate it. A stream worker reads continuously and writes every few seconds.

Does PipeKeel support Confluent Cloud and Aiven?

Yes — Confluent with SASL/PLAIN (API key and secret), Aiven with client certificates or SASL, Amazon MSK with SASL/SCRAM, and self-hosted clusters.

Start with the Team plan, free for 14 days

Then keep a free plan, or choose a paid one. No credit card.

Start free All connectors