A Clojure benchmarking thing

I needed a tool to evaluate internal and network benchmarks of Riemann, to ask questions like

  • Is parser function A or B more efficient?
  • How many threads should I allocate to the worker threadpool?
  • How did commit 2556 impact the latency distribution?

In dealing with “realtime” systems it's often a lot more important to understand the latency distribution rather than a single throughput figure, and for GC reasons you often want to see a time dependence. Basho Bench does this well, but it's in Erlang which rules out microbenchmarking of Riemann functions (e.g. at the repl). So I've hacked together this little thing I'm calling Schadenfreude (from German; “happiness at the misfortune of others”). Sums up how I feel about benchmarks in general.

; A run is a benchmark specification. :f is the function we're going to ; measure--in this case, counting using ; ; 1. an atomic reference ; 2. unordered (commute) transactions ; 3. ordered (alter) transactions. ; ; :before and :after are callbacks to set up and tear down for the test run. (let [runs [(let [a (atom 0)] {:name "atoms" :before #(reset! a 0) :f #(swap! a inc)}) (let [r (ref 0)] {:name "commute" :before #(dosync (ref-set r 0)) :f #(dosync (commute r inc))}) (let [r (ref 0)] {:name "alter" :before #(dosync (ref-set r 0)) :f #(dosync (alter r inc))})] ; For these benchmarks, we'll prime the JVM by doing the test twice and ; discarding the first one's results. We'll run each benchmark 10K times runs (map #(merge % {:prime true :n 10000}) runs) ; And we'll try each one with 1 and 2 threads runs (mapcat (fn [run] (map (fn [threads] (merge run {:threads threads :name (str (:name run) " " threads)})) [1 2])) runs) ; Actually run the function and collect data runs (map record runs) ; And plot the results together plot (latency-plot runs)] ; For this one we'll use a log plot. (.setRangeAxis (.getPlot plot) (org.jfree.chart.axis.LogarithmicAxis. "Latency (s)")) (view plot))

latency.png

When I have something usable outside a REPL I'll publish it to clojars and github. Right now I think the time alignment looks pretty dodgy so I'd like to normalize it correctly, and figure out what exactly “throughput” means. Oh, and the actual timing code is completely naive: no OS cache drop, no forced GC/finalizers, etc. I'm gonna look into tapping Criterium's code for that.

Post a Comment

Please avoid writing anything here unless you are a computer: This is also a trap:

Supports github-flavored markdown for [links](http://foo.com/), *emphasis*, _underline_, `code`, and > blockquotes. Use ```clj on its own line to start a Clojure code block, and ``` to end the block.

Copyright © 2014 Kyle Kingsbury.
Non-commercial re-use with attribution encouraged; all other rights reserved.
Comments are the property of respective posters.