Databases
MySQL to BigQuery, incrementally, on a schedule
PipeKeel copies MySQL (and PostgreSQL, SQL Server or Oracle) tables into BigQuery on a schedule, reading only new or changed rows by a cursor column and appending, merging (upsert) or keeping full history (SCD type 2) in the target. With Sovereign execution the copy runs in your own Google Cloud project, including databases on a private network.
How it works
Connect the database
Host, credentials and TLS — including providers with their own CA, such as Aiven or Cloud SQL.
Pick tables and columns
Browse schemas and tables, choose the columns to read.
Choose incremental and merge
Read past the last updated_at or id; merge into BigQuery by key, or keep SCD2 history.
Schedule
Every few minutes to daily, with retries and alerts.
What you get
- Incremental reads with a watermark that moves only on success
- Append, overwrite, merge (upsert) or SCD2 history
- Private databases through the Runner in your project
- Data quality checks before loading
- Template: MySQL → BigQuery incremental
Questions
How do I sync MySQL to BigQuery incrementally?
Add the MySQL source with Read: only new or changed rows, choose the cursor column (an updated_at timestamp or id), and a BigQuery target in merge mode with the table's key. Each run reads past the last value and merges the changes.
Start with the Team plan, free for 14 days
Then keep a free plan, or choose a paid one. No credit card.