Ismail Iusifli
← All work
Infrastructure Utility2026 · Author

pg-interleave: Deterministic Race-Condition Tests for Postgres

Wire-protocol proxy · Statement barriers · Torn-commit faults · Seeded schedule fuzzer · Open source

View on GitHub
Lost updates found, 5 ms offset
0–5 → 85–94 / 100
Plain run vs seeded fuzzer, two machines. A toy single-row scenario, not a claim about real services
Proxy overhead
~0.2 ms / query
Measured on a Linux box after fixing a 44 ms Nagle and delayed-ACK artifact with TCP_NODELAY
Run against pg-boss v12.38.1
8 tests, 0 bugs
A negative result. The bug it did find was in the tool's own regex matching

Key decisions

A proxy that speaks the PostgreSQL wire protocol

RejectedTCP-level fault injection such as Toxiproxy

At the TCP layer a SELECT and a COMMIT look the same. Understanding the protocol lets a test hold one specific statement while other connections keep running, with no change to application code beyond the database URL.

Barriers for known interleavings, plus a seeded fuzzer for unknown ones

RejectedTests that start both transactions together, as a Promise.all test does

Started together, the plain run finds a lost update almost every time. A few milliseconds apart it finds it 0 to 5 times in 100, because the race window is narrow and depends on the machine. The fuzzer adds random delays so the transactions overlap anyway.

Every rule handle has assertFired(n)

RejectedRules that silently fail to match and let the query through

My first pg-boss deadlock test passed because the single-line regex never matched pg-boss's multi-line statement, so no fault was injected. A rule that does not fire looks exactly like a system that handled the fault, so the test now fails if the fault never happened.

Context

A green concurrency test only shows that the test did not reach the race. In a one-row lost-update experiment the window was a few milliseconds wide, and it moved between my two machines. pg-interleave is a proxy that sits between your code and Postgres so a test can control the order in which statements arrive, instead of hoping the timing lines up.

It is alpha software under the MIT license, published to npm as pg-interleave (latest 0.7.1). The server is written in Rust on tokio and the client is a TypeScript SDK. You point the database URL at the proxy's port, and the test drives it over a small HTTP API.

What it does

  • Statement barriers. Hold one statement in the proxy, let another connection run a whole transaction, then release. The lost update becomes a deterministic test with no sleep.
  • Torn commits. With drop_connection_on_commit the commit finishes on the server and the client's socket is then cut. A generic retry helper tries again, and a transaction without an idempotency key inserts twice. A mocked connection cannot show that, because a mocked drop leaves no committed row behind.
  • Schedule fuzzer. Random delays before queries, including BEGIN and COMMIT. Each connection gets its own stream derived from a seed with SplitMix64. Client-side timing still varies by a few milliseconds, so a seed gives a close result, not an identical one.

Testing it on software I did not write

I wrote eight tests for pg-boss v12.38.1, a job queue on top of Postgres: an injected 40P01 deadlock in the fetch statement, a torn commit on job completion, a held completion statement, a handler that outlives its expiry, and bursts of concurrent send() calls. None of them found a bug in pg-boss. The expiry test showed pg-boss's documented at-least-once behavior.

The bug in my own tool

My first deadlock test passed, and it should not have. pg-boss writes its fetch statement across several lines, my pattern was a single-line regex, and in a regex . does not cross a newline. The rule never matched, the proxy forwarded the query untouched, and the worker behaved normally. A reviewer found it by running my pattern against the real statement.

Two changes came out of it: the SDK now compiles a RegExp with the dotAll flag and keeps i and m, and every rule handle has assertFired(n), which throws if the rule did not fire exactly n times.

The 44 ms trap

An early version of my README claimed that the fuzzer finds 100% of lost updates. That number was an artifact. The proxy added about 44 ms to every query, which is consistent with Nagle's algorithm interacting with delayed ACKs. With queries that slow, the two transactions overlapped whether or not the fuzzer was on, so the comparison meant nothing. Setting TCP_NODELAY on both sockets dropped the overhead to about 0.2 ms per query, and the plain baseline at a 5 ms offset fell to between 0 and 5 out of 100. I corrected the README and wrote the story up.

Limitations

  • Plaintext only. The proxy answers N to SSLRequest, so sslmode=require does not work.
  • Flush pipelines. Drivers that rely on explicit Flush frames before Sync can stall the proxy.
  • Data-modifying CTEs. The AST selector classifies WITH u AS (UPDATE ...) SELECT ... as a SELECT. Use a regex for those.
  • PgBouncer in transaction mode. Backend PIDs change between transactions, so route rules with SQL comment tags.
  • Shared state and no auth. Rules live in the proxy process and the control plane has no authentication. Run test files serially and keep it on 127.0.0.1.
  • One toy benchmark. The race window was a different width on my two machines, and I have not measured the fuzzer on a large real application.

When not to use it

If you can write the scenario as plain sequential awaits on two clients, you do not need a proxy. PostgreSQL's own isolationtester is a better fit when you are testing the database engine. Toxiproxy is the right layer for latency and dropped connections. pg-interleave is for the case in between: application code you do not want to modify, where you want to stop one specific statement.

Next caseSchool Platform: Landing + CRM + LMS Work togetherNeed this kind of backend? Email me