Change data capture (CDC)

Without CDC, every run would resend your entire dataset, which means more API calls, slower syncs, and wasted downstream processing. With CDC, a sync sends only what changed since the last run.

​

Before you begin

• A model with a unique primary key.

​

How CDC works

Each time a sync runs, Hightouch compares the current query results against a stored snapshot of the previous run, called the diff file, and sends only the rows that differ:

1. Query the model against your source.
2. Compare the results to the previous run’s diff file, matching rows by primary key.
3. Classify each row as added, changed, or removed.
4. Send those changes to the destination, along with any rows that failed on the previous run.
5. Save a new diff file for the next comparison.

​

CDC depends on a unique primary key

The primary key is how CDC identifies a row from one run to the next. If primary keys are missing or duplicated, Hightouch can’t reliably tell an added row from a changed one. See primary key requirements.

​

CDC by sync mode

The sync mode determines which detected changes become destination operations:

• Insert — sends only rows whose primary key is new since the previous run.
• Update and Upsert — send rows whose mapped values changed.
• Archive and All (sometimes called Mirror) — don’t use CDC. Every row is sent on every run, and these modes don’t store requests in the live debugger.

For the full mode reference, see sync modes and change data capture.

​

Reset or reprocess CDC

Some changes require Hightouch to rebuild CDC state before the next run behaves as you expect. Reset CDC, full resync, and clear-and-fill are run behaviors, not sync modes. For the consequences and click paths of each action, see the Syncs overview.

​

When CDC runs

CDC happens after Hightouch receives the query results and before it writes to the destination. While the sync status shows Querying, Hightouch is waiting for query results, computing the diff, or saving the new diff file.

​

Where CDC runs

By default, Hightouch computes CDC on its own infrastructure, called the basic engine. For supported sources, the Lightning Sync Engine computes CDC directly in your warehouse, which speeds up syncs at higher volumes.

​

Where diff files are stored

Hightouch stores the previous run’s query results to compute the next diff. By default, diff files live in an encrypted, Hightouch-managed bucket and are retained for 30 days. Some plans let you bring your own bucket to keep diffs in your own infrastructure. If a sync doesn’t run for more than 30 days, its diff file expires. The next run needs a reset CDC or a full resync to establish a new baseline.

​

Why reverse ETL uses difference-based CDC

Hightouch compares full before-and-after query results, a method called difference-based CDC. Warehouses can’t produce CDC logs for arbitrary SQL queries or dbt models, so log-based CDC isn’t possible when reading from them. This is the reverse of traditional ETL. OLTP databases such as Postgres and MySQL log incremental changes natively, and ETL tools use those logs when loading data into a warehouse. Because Hightouch sends data out of the warehouse, it diffs query results instead.

​

FAQ

Does Hightouch keep historical diff files? No. Hightouch stores only the most recent diff file and compares against it. What happens if a sync fails? CDC retries the failed rows on the next run. See retries. What happens if I stop syncing for more than 30 days? The diff file in managed storage expires. Reset CDC or run a full resync to rebuild the baseline. What happens if I change my mappings? Changing mappings can trigger a full reprocess on the next run. See field mapping updates.