When I designed UState, I had a goal of a thousand state transitions per second. I hit about six hundred on my Macbook Pro, and skirted 1000/s on real hardware. Eventmachine is good, but I started to bump up against concurrency limits in MRI’s interpreter lock, my ability to generate and exchange SQL with SQLite, and protobuf parse times. So I set out to write a faster server. I chose Clojure for its expressiveness and powerful model of concurrent state–and more importantly, the JVM, which gets me Netty, a mature virtual machine with a decent thread model, and a wealth of fast libraries for parsing, state, and statistics. That project is called Riemann.

Today, I’m pleased to announce that Riemann crossed the 10,000 event/second mark in production. In fact it’s skirting 11k in my stress tests. (That final drop in throughput is an artifact of the graph system showing partially-complete data.)

By the way, we push about 200 events/sec through a single Riemann server from all of Showyou’s infrastructure. There’s a lot of headroom.

I did the dumbest, easiest things possible. No profiling. A heavy abstraction (aleph) on top of netty. I haven’t even turned on warn-on-reflection or provided type hints yet. All operations are over synchronous TCP. This benchmark measures Riemann’s ability to thread events through a complex set of streams including dozens of (where) filters and updating the index with every received event.

I’m in the final stages of packaging Riemann for initial public release this week. Boundary has also kindly volunteered their space for a tech talk on Riemann: Thursday, March 1st, at Boundary’s offices, likely at 7 PM. I’ll post a Meetup link here and on Twitter shortly.