A lot of my work involves staring at visualizations trying to get an intuitive feeling for what a system is doing. I’ve been working on a new visualization for Jepsen, a distributed systems testing library. This is something I’ve had in the back of my head for years but never quite got around to.

A Jepsen test records a history of operations. Those operations often come in a few different flavors. For instance, if we’re testing a queue, we might send messages into the queue, and try to read them back at the end. It would be bad if some messages didn’t come back; that could mean data loss. It would also be bad if messages came out that were never enqueued; that could signify data corruption. A Jepsen checker for a queue might build up some data structures with statistics and examples of these different flavors: which records were lost, unexpected, and so on. Here’s an example from the NATS test I’ve been working on this month:

{:valid?             false,
 :attempt-count      529583,
 :acknowledged-count 529369,
 :read-count         242123,
 :ok-count           242123,
 :recovered-count    3
 :hole-count         159427,
 :lost-count         287249,
 :unexpected-count   0,
 :lost               #{"110-6014" ... "86-8234"},
 :holes              #{"110-4072" ...  "86-8234"},
 :unexpected         #{}}