nicolas-ha.com · Français

Moving a bank's core without stopping production.

Griffin Bank, 2022 to 2025, while it obtained its banking licence. The message queue at the centre of the bank, in three generations.

The problem: the bank is a message system. Its central queue had to move from Kafka to FoundationDB, with the bank running.

1The diagram

The procs, the runner and the queue on FoundationDB, Kafka before; the audit log and the distributed cron; three generations
Drawn in Excalidraw.
the business codethe runner and its generationsthe queuebuilt on top

2What I built

  1. The runner: the bank's "procs" are pure functions, every message declared in clojure.spec; I wrote the runner in every generation.
  2. v1, the migration: starting from the CTO's proof of concept, I wrote the runners and all the migration tooling, then ran the migration.
  3. v2 and v3, rewritten from scratch: concurrency, then fairness and isolation between workloads.
    • From Dynamo (Amazon, SOSP 2007: the consistent-hashing ring) and QuiCK (Apple, SIGMOD 2021).
  4. The distributed cron: built on the queue, each scheduled job runs exactly once in the cluster.
  5. The audit log: it indexes every message; the list of specs is the list of what it indexes.

3The result

  1. Zero errors, zero incidents at the switchover.
    • So quiet I first thought the switch had not been flipped; the VP called it out at an all-hands.
  2. A third of the audit store saved, by making its history optional.
  3. Generated tests: a framework on clojure.spec and test.check, in FoundationDB's lineage, that finds bugs fast.

4My role

  1. Senior software engineer, on the platform.
    • I helped stand up new teams as the bank grew.
  2. Internal tech talks, RFCs and mentoring: message queues and fairness, distributed crons.

5The video

One minute, synthetic voice, subtitled.