Jepsen is a library for writing tests of concurrent systems: everything from single-node data structures to distributed databases and queues. A key part of this process is recording a history of operations performed during the test. Jepsen checkers analyze a history to find consistency anomalies and to compute performance metrics. Traditionally Jepsen has stored the history in a Clojure vector (an immutable in-memory data structure like an array), and serialized it to disk at the end of the test. This limited Jepsen to histories on the order of tens of millions of operations. It also meant that if Jepsen crashed during a several-hour test run, it was impossible to recover any of the history for analysis. Finally, saving and loading large tests involved long wait times—sometimes upwards of ten minutes.
Over the last year I’ve been working on ways to resolve these problems. Generators are up to ten times faster. A new operation datatype makes each operation smaller and faster to access. Jepsen’s new on-disk format allows us to stream histories incrementally to disk, to work with histories of up to a billion operations far exceeding available memory, to recover safely from crashes, and to load tests almost instantly by deserializing data lazily. New history datatypes support both densely and sparsely indexed histories, and efficiently cache auxiliary indices. They also support lazy disk-backed map and filter. These histories support both linear and concurrent folds, which dramatically improves checker performance on multicore systems: real-world checkers can readily analyze 250,000 operations/sec. Histories support multi-query optimization: when multiple threads fold over the same history, a query planner automatically fuses those folds together to perform them in a single pass. Since Jepsen often folds dozens of times over the same history, this saves a good deal of disk IO and deserialization time. These features are enabled by a new, transactional, dependency-aware task executor.
Again with the reductions! I keep writing code which reduces over a collection, keeping track of more than one variable. For instance, here’s one way to find the mean of a collection of integers:
(defn mean
"A reducer to find the mean of a collection. Accumulators are [sum count] pairs."
([] [0 0])
([[sum count]] (/ sum count))
([[sum count] x]
[(+ sum x) (inc count)]))
This mean function is what Clojure calls a reducer, or a reducing function. With no arguments, it constructs a fresh accumulator. With two arguments, it combines an element of some collection with the accumulator, returning a new accumulator. With one argument, it transforms the accumulator into some final result.
Update 2022-08-12: The Hamilton County Health Department now has a page about monkeypox with symptoms and isolation guidance, as well as options for vaccination, testing, and treatment–look for “complete our monkeypox vaccine registration”. The Cincinnati Health Department is also offering vaccines for high-risk groups. People in Hamilton County without a primary care physician who have symptoms can also call call 513-357-7320 for the Cincinnati city health clinic.
If you’re a gregarious gay man like me you’ve probably heard about monkeypox. Monkeypox is an orthopoxvirus which causes, in addition to systemic symptoms, lesions on the skin and mucosa. It’s transmitted primarily through skin-to-skin contact, though close-range droplet and fomite transfer are also possible. The current outbreak in industrialized nations is almost entirely among gay, bisexual, and other men who have sex with men (GBMSM); likely via sexual networks. In the UK, for example, 99% of cases are male and 97% are among GBMSM. Ontario reports 99% of cases are in men. In New York 99% of cases are in men who have sex with men. For a good overview of what monkeypox looks like, how it’s spread, and ways we can reduce transmission, check out San Francisco Leathermen’s Discussion Group’s presentation by MPH Frank Strona.
I write a lot of reductions: loops that combine every element from a collection in some way. For example, summing a vector of integers:
(reduce (fn [sum x] (+ sum x)) 0 [1 2 3])
; => 6
If you’re not familiar with Clojure’s reduce, it takes a reducing function f, an initial accumulator init, and a collection xs. It then invokes (f init x0) where x0 is the first element in xs. f returns a new accumulator value acc1, which is then passed to (f acc1 x1) to produce a new accumulator acc2, and so on until every x in xs is folded into the accumulator. That accumulator is the return value of reduce.
In writing reductions, there are some problems that I run into over and over. For example, what if you want to find the mean of some numbers in a single pass? You need two accumulator variables–a sum and a count. The usual answer to this is to make the accumulator a vector tuple. Destructuring bind makes this… not totally awful, but a little awkward:
(reduce (fn [[sum count] x]
[(+ sum x) (inc count)])
[0 0]
[1 2 3 4 5 6 7])
; => [28 7]
This history is also available as a PDF or EPUB, which may be more pleasant for reading.
The argument goes like this.
Kink, leather, and BDSM do not belong at Pride. First, they aren’t actually LGBTQ: kink is also practiced by straight people (Baker-Jordan, 2021). Moreover, those queer people who do display kink at Pride expose vulnerable people to harmful symbols and acts. They wear pup hoods and rubber bodices, they dress in studded codpieces and leather harnesses, they sport floggers, handcuffs, and nipple clamps (lesbiansofpower, 2021; stellar_seabass, 2021). Some demonstrate kinky acts: they crack whips in the parade and chain themselves up on floats. Some have sex in public (kidpiratez, 2021).
These displays harm three classes of people. Children (and the larger class of minors, e.g. those under 18 or 21) are innocent and lack the sophistication to process what they are seeing: exposure to kink might frighten them or distort their normal development (Angel, 2021; Barrie, 2021). Asexual people, especially those who are sex-repulsed, may suffer emotional harm by being confronted with overt displays of sexuality (Dusty, 2021; roseburgmelissa, 2021). Finally, those with trauma may be triggered by these displays (stymstem, 2021). These hazards exclude vulnerable people from attending Pride: kink is therefore a barrier to accessibility (RiLo_10, 2021; Vaush, 2021).
Consent is key to healthy BDSM practice, but the public did not consent to seeing these sexual displays (Baker-Jordan, 2021; busytoebeans, 2021; prettycringey, 2021). By wearing leather harnesses and chaining each other up in broad daylight, kinksters have unethically involved non-consenting bystanders in a BDSM scene for their own (likely sexual) gratification (anemersi, 2021; Bartosch, 2020; Xavier’s Online, 2021a, 2021b). The lack of consent to these sexual displays constitutes a form of sexual assault (PencilApocalyps, 2021). At worst, the fact that children may be present in the crowd makes these displays pedophilia (Rose, 2021), and (if one is so inclined) exemplifies the moral degeneracy of the entire LGBTQ community and impending collapse of civilization (Dreher, 2021; Keki, 2019)1.
Not everyone holds all of these views, or holds them to this degree; this is a synthesis of one pole in a diverse and vigorous debate. Nevertheless, calls to ban kink at Pride remain a mainstay of Twitter and Tumblr every June. To some extent this position is advanced by anti-gay reactionaries on 4chan and Telegram channels (Piper, 2021), but this is not the whole story: many opposed to kink at Pride identify themselves as queer, or at least queer-friendly (Mahale, 2021).
Hey y’all! It’s been, gosh, what, ten years? I finally finished a total site redesign: all-new backend, HTML, CSS, modern image formats, etc. It’s finally readable on mobile now!
There’s a lot of accumulated cruft in the database and filesystem–aphyr.com is old enough that it still has redirects for CGI scripts written circa 2005. While I’ve tried as hard as I can to preserve compatibility, older posts may not look great, or there might be subtle formatting/text-processing issues. If you notice anything that looks super broken, leave a comment (either on the post itself or here), and I’ll try to get it sorted out!
Hey y’all. Doing some long-overdue upgrades on aphyr.com; service will be up and down for a few hours; emails might bounce, etc. as I get things sorted.
Update: All finished, thanks for bearing with me!
Previously: Rewriting the Technical Interview.
Aisha’s hands rattle you. They float gently in front of her shoulders, wrists cocked back. One sways cheerfully as she banters with the hiring manager—her lacquered nails a cyan mosaic over ochre palms. They flit, then hover momentarily as the two women arrange lunch. When the door closes, Aisha slaps her fingertips eagerly on the pine-veneer tabletop. Where have you seen them before?
Previously: Typing the Technical Interview.
Update, November 2023: here are the full term rewrite and language macros which formed the seed of this story. These files include OO notation as well as the basic Algol syntax shown here. There is also a sketch of an object-oriented language with classes and inheritance, implemented as a Clojure macro. I do not remember writing it. It looks terrifying.
Previously: Debugging.
In this chapter, we’ll discuss some of Clojure’s mechanisms for polymorphism: writing programs that do different things depending on what kind of inputs they receive. We’ll show ways to write open functions, which can be extended to new conditions later on, without changing their original definitions. Along the way, we’ll investigate Clojure’s type system in more detail–discussing interfaces, protocols, how to construct our own datatypes, and the relationships between types which let us write flexible programs.
Update 2022-03-31: Owen Jacobson has written a lovely companion to this piece which covers new dynamics in update 5.
With apologies, as usual, to Christopher Alexander.
Stay home.
I’ve been talking to folks 1:1 about this, but from a scroll through the feed today, I don’t think the general community has caught on. COVID-19 is not fucking around. If we don’t contain or dramatically slow it, we are going to run out of health care workers, hospital beds, and equipment. People are going to die for want of care. This is not a problem of the distant future: recent modeling suggests that without a significant reduction in social contact, Seattle will exhaust healthcare capacity around two weeks from now. Other regions will not be far behind.