PipeKeelBeta

Streaming

Pub/Sub to BigQuery in real time, transformed on the way

PipeKeel turns a Pub/Sub → BigQuery pipeline into a native BigQuery subscription in your Google Cloud project, compiling the steps in between — select, filter, derive, change types, mask, route — into a Pub/Sub message transform. Rows land within seconds and nothing runs on PipeKeel or on a Dataflow job you'd have to manage and pay for.

Start free See pricing

How it works

  1. Set up Sovereign execution

    Run PipeKeel's setup script in your project with real-time streams turned on.

  2. Describe the messages

    List the JSON fields to read into typed columns, plus message id, publish time and attributes if you like.

  3. Add steps and BigQuery targets

    Filter, derive, mask personal data, route to several tables.

  4. Activate

    PipeKeel creates the subscriptions and keeps them in step with the pipeline — pause it and they're removed.

What you get

  • Native BigQuery subscriptions: seconds of latency, no servers
  • Steps compiled to a Pub/Sub JavaScript transform, with SQL semantics
  • Masking and hashing personal data before it lands
  • Micro-batches in Express for joins, aggregates and any target
  • Publish rows back to Pub/Sub topics from any pipeline

Questions

Can I transform Pub/Sub messages before they reach BigQuery?

Yes. PipeKeel compiles filters, derived columns, type changes, masking and routing into a single message transform on the BigQuery subscription, so each message is transformed as it's written.

Do I need Dataflow?

No. Real-time Pub/Sub uses BigQuery subscriptions with message transforms in your project; there is no streaming job to run or pay for.

Start with the Team plan, free for 14 days

Then keep a free plan, or choose a paid one. No credit card.

Start free All connectors