Riemann: Breaking the 10k barrier

When I designed UState, I had a goal of a thousand state transitions per second. I hit about six hundred on my Macbook Pro, and skirted 1000/s on real hardware. Eventmachine is good, but I started to bump up against concurrency limits in MRI's interpreter lock, my ability to generate and exchange SQL with SQLite, and protobuf parse times. So I set out to write a faster server. I chose Clojure for its expressiveness and powerful model of concurrent state–and more importantly, the JVM, which gets me Netty, a mature virtual machine with a decent thread model, and a wealth of fast libraries for parsing, state, and statistics. That project is called Riemann.

Today, I'm pleased to announce that Riemann crossed the 10,000 event/second mark in production. In fact it's skirting 11k in my stress tests. (That final drop in throughput is an artifact of the graph system showing partially-complete data.)

throughput.png

cpu.png

By the way, we push about 200 events/sec through a single Riemann server from all of Showyou's infrastructure. There's a lot of headroom.

I did the dumbest, easiest things possible. No profiling. A heavy abstraction (aleph) on top of netty. I haven't even turned on warn-on-reflection or provided type hints yet. All operations are over synchronous TCP. This benchmark measures Riemann's ability to thread events through a complex set of streams including dozens of (where) filters and updating the index with every received event.

10k.png

I'm in the final stages of packaging Riemann for initial public release this week. Boundary has also kindly volunteered their space for a tech talk on Riemann: Thursday, March 1st, at Boundary's offices, likely at 7 PM. I'll post a Meetup link here and on Twitter shortly.

Tim
Tim, on

Looking forward to that tech talk!

Post a Comment

Please avoid writing anything here unless you are a computer: This is also a trap:

Supports github-flavored markdown for [links](http://foo.com/), *emphasis*, _underline_, `code`, and > blockquotes. Use ```clj on its own line to start a Clojure code block, and ``` to end the block.

Copyright © 2017 Kyle Kingsbury.
Non-commercial re-use with attribution encouraged; all other rights reserved.
Comments are the property of respective posters.