From reading WAL to moving data

I have been obsessed with PostgreSQL internals for a while: WAL, replication slots, MVCC, and logical decoding.

I built pgstream first. It could read PostgreSQL WAL changes and stream them in real time. That project got attention I genuinely did not expect, but it was still mainly a reader. It could show changes, but it did not give those changes anywhere useful to go.

So I started building Rift.

The idea was simple:

A single Go binary that reads PostgreSQL changes and ships them where they need to go.

A webhook. Another PostgreSQL database. Redis. No Kafka cluster, no ZooKeeper, no Debezium setup, and no JVM infrastructure for a small pipeline.

Why not just use Debezium?

Debezium is excellent. It is battle-tested and widely used in serious production systems.

But spinning up Kafka Connect to sync a few tables can be excessive for a side project, a small team, or a focused production workflow. I wanted something that could be installed, configured, and running quickly.

Rift is built for that gap.

It currently:

  • Reads PostgreSQL WAL through logical replication.
  • Decodes INSERT, UPDATE, and DELETE events.
  • Filters or transforms events with JavaScript through Goja.
  • Sends events to webhooks, PostgreSQL, or Redis.
  • Queues failed deliveries to disk with BoltDB instead of dropping them.

That last part became much more important once I started breaking things deliberately.

A destination being offline should not destroy data

A CDC pipeline is only useful if it survives the moments when another system does not.

If a webhook endpoint, Redis instance, or destination database is temporarily unavailable, Rift stores events in an embedded disk queue. The pipeline can retry later instead of blocking the stream forever or losing changes.

That sounds obvious in theory. It becomes very real when a failed event keeps returning every few seconds.

Which brings me to the bug that consumed my weekend.

The DDL bug that kept retrying forever

After INSERT, UPDATE, and DELETE were working, I wanted Rift to notice schema changes too.

For example, if someone runs:

ALTER TABLE users ADD COLUMN city TEXT;

Rift should know that the source schema changed instead of silently failing the next time a new row arrives.

My approach used a PostgreSQL event trigger:

  • Rift installs a function and event trigger in the source database.
  • DDL commands such as CREATE TABLE, ALTER TABLE, and CREATE INDEX trigger it.
  • The trigger writes a record into an internal table named rift_ddl_log.
  • Because that table is normal PostgreSQL data, its INSERT events also appear in the WAL stream.
  • Rift reads those events and forwards them as DDL metadata.

It sounded clean.

Then Rift started trying to replicate its own internal rift_ddl_log rows into destination databases that did not have a rift_ddl_log table.

The result was an infinite loop:

ERROR: relation "rift_ddl_log" does not exist

The failed event went into the disk queue, got retried, failed again, and repeated forever.

The bug was not the condition. It was the data shape.

My first fix was a strict table-name check.

That failed because some replication paths sent the table name as:

rift_ddl_log

And other paths sent:

public.rift_ddl_log

I was staring at the condition instead of logging the actual value moving through the system.

The proper fix became a helper that handles schema-qualified and unqualified internal table names:

func isInternalTable(table string) bool {
    internal := []string{"rift_ddl_log", "public.rift_ddl_log"}

    for _, t := range internal {
        if strings.EqualFold(table, t) || strings.HasSuffix(table, t) {
            return true
        }
    }

    return false
}

After that, internal log events were filtered cleanly while real schema-change events still flowed through the pipeline.

The lesson was bigger than this one bug:

Before fixing a condition, inspect the real shape of the data reaching it.

DDL tracking is not schema replication

This work exposed another design boundary.

Rift can detect that someone ran CREATE TABLE users. But that does not mean Rift should automatically run the same CREATE TABLE command in every destination.

Those are two separate problems.

Tracking DDL means Rift can report that a schema changed. Full schema replication means safely applying that change elsewhere, dealing with types, permissions, migration order, existing data, and conflicts.

Rift does the first today. The second is intentionally not promised yet.

I would rather ship a tool with a clear boundary than pretend it solves a harder problem than it actually does.

Making Rift feel like a real tool

At first, I ran everything through:

go run main.go

That is fine for experimentation. It is not how you want people to use a project.

So I rebuilt the CLI with Cobra. Rift now has an actual command interface:

rift run
rift run -c custom-config.yaml
rift version

That sounds like a small change, but it changed the whole feeling of the project. It stopped being a repository you clone to inspect and became a tool you can install and use.

Where Rift is now

Rift can currently:

  • Stream INSERT, UPDATE, and DELETE changes from PostgreSQL.
  • Deliver events to webhooks, PostgreSQL, or Redis.
  • Filter and transform events with JavaScript.
  • Track DDL changes through PostgreSQL event triggers.
  • Persist failed deliveries with an embedded disk queue.
  • Run as an installable Go CLI.

It is still early, and that is exactly why I want people to break it.

The fastest way to learn what a system needs is to see where real users push it beyond your assumptions.

Source: github.com/mujib77/rift