Riemann 0.2.0 is ready. There’s so much left that I want to build, but this release includes a ton of changes that should improve usability for everyone, and I’m excited to announce its release.

Version 0.2.0 is a fairly major improvement in Riemann’s performance and capabilities. Many things have been solidified, expanded, or tuned, and there are a few completely new ideas as well. There are a few minor API changes, mostly to internal structure–but a few streams are involved as well. Most functions will continue to work normally, but log a deprecation notice when used.

I dedicated the past six months to working on Riemann full-time. I was fortunate to receive individual donations as well as formal contracts with Blue Mountain Capital, SevenScale, and Iovation during that time. That money gave me months of runway to help make these improvements–but even more valuable was the feedback I received from production users, big and small. I’ve used your complaints, frustrations, and ideas to plan Riemann’s roadmap, and I hope this release reflects that.

This release includes contributions from a broad cohort of open-source developers, and I want to recognize everyone who volunteered their time and energy to make Riemann better. In particular, I’d like to call out Pierre-Yves Ritschard, lwf, Ben Black, Thomas Omans, Dave Cottlehuber, and, well, the list goes on and on. You rock.

These months have seen not only improvements to Riemann itself, but to the dashboard, clients, and integration packages. While I’m spending most of my time working on the core Riemann server, it’s really this peripheral software that make Riemann useful for instrumenting production systems. There’s no way I could hope to understand, let alone write and test the code to integrate with all these technologies–which makes your work particularly valuable.

This week I started my new job at Factual. I won’t be able to work 10 hours each day on Riemann any more, but I’m really happy with what we’ve built together, and I’ll definitely keep working on the next release.

To all Riemann’s users and contributors, thank you. Here’s to 0.2.0.

New features

• Arbitrary key-value (string) pairs on events
• Hot config reloading
• Integrated nrepl server
• streams/sdo: bind together multiple streams as one
• streams/split: like (cond), dispatch an event to the first matching stream
• streams/splitp: like split, but on the basis of a specific predicate
• config/delete-from-index: explicitly remove (similar) events from the index
• streams/top: streaming top-k
• streams/tag: add tags to events
• RPM packaging
• Init scripts, proper log dirs, and users for debian and RPM packages. Yeah, this means you can /etc/init.d/riemann reload, and Stuff Just Works (TM).
• folds/difference, product, and quotient.
• Folds come in sloppy and strict variants which should “Do What I Mean” in most contexts.
• Executor Services for asynchronous queued processing of events.
• streams/exception-stream: captures exceptions and converts them to events.

Improvements

• http://riemann.io site
• Lots more documentation and examples
• Config file syntax errors are detected early
• Cleaned up server logging
• Helpful messages (line numbers! filenames!) for configuration errors
• Silence closed channel exceptions
• Cores can preserve services like pubsub, the index, etc through reloads
• Massive speedups in TCP and UDP server throughput
• streams/rate works in real-time: no need for fill-in any more
• Graphite client is faster, more complete
• Config files can include other files by relative path
• streams/coalesce passes on expired events
• riemann.email/mailer can take custom :subject and :body functions
• riemann.config includes some common time/scheduling functions
• streams/where returns whether it matched an event, which means (where) is now re-usable as a predicate in lots of different contexts.
• streams/tagged-any and tagged-all return whether they matched
• streams/counter is resettable to a particular metric, and supports expiry
• Bring back “hyperspace core online”
• Update to netty 3.6.1
• Reduced the number of threadpools used by the servers
• Massive speedup in Netty performance by re-organizing execution handlers
• core/reaper takes a :keep-keys option to specify which fields on an event are preserved
• streams/smap ignores nil values for better use with folds
• Update to aleph 0.3.0-beta15
• Config files ship with emacs modelines, too

Bugfixes

• Fixed a bug in part-time-fast causing undercounting under high contention
• Catch exceptions while processing expired events
• Fix a bug escaping metric names for librato
• riemann.email/mailer can talk to SMTP relays again
• graphite-path-percentiles will convert decimals of three or more places to percentile strings
• streams/rollup is much more efficient; doesn’t leak tasks
• streams/rollup aggregates and forwards expired events instead of stopping
• Fixed a threadpool leak from Netty
• streams/coalesce: fixed a bug involving lazy persistence of transients
• streams/ddt: fixed a few edge cases

Internals

• Cleaned up the test suite’s logging
• Pluggable transports for netty servers
• Cores are immutable
• Service protocol: provides lifecycle management for internal components
• Tests for riemann.config
• riemann.periodic is gone; replaced by riemann.time
• Tried to clean up some duplicated functions between core, config, and streams
• riemann.common/deprecated
• Cleaned up riemann.streams, removing unused commented-out code
• Lots of anonymous functions have names now, to help with profiling
• Composing netty pipeline factories is much simpler
• Clojure 1.5

Known bugs

• Passing :host to websocket-server does nothing: it binds to * regardless.
• Folds/mean throws when it receives empty lists
• graphite-server has no tests
• Riemann will happily overload browsers via websockets
• streams/rate doesn’t stop its internal poller correctly when self-expiring
• When Netty runs out of filehandles, it’ll hang new connections