A good friend of mine from college has started teaching himself to code. He’s hoping to find a job at a Bay Area startup, and asked for some help getting oriented. I started writing a response, and it got a little out of hand. Figure this might be of interest for somebody else on this path. :)
I want to give you a larger context around how this field works–there’s a ton of good documentation on accomplishing specifics, but it’s hard to know how it fits together, sometimes. Might be interesting for you to skim this before we meet tomorrow, so some of the concepts will be familiar.
Schadenfreude is a benchmarking tool I’m using to improve Riemann. Here’s a profile generated by the new riemann-bench, comparing a few recent releases in their single-threaded TCP server throughput. These results are dominated by loopback read latency–maxing out at about 8-9 kiloevents/sec. I’ll be using schadenfreude to improve client performance in high-volume and multicore scenarios.
I needed a tool to evaluate internal and network benchmarks of Riemann, to ask questions like
- Is parser function A or B more efficient?
- How many threads should I allocate to the worker threadpool?
- How did commit 2556 impact the latency distribution?
Went digging through the FBI’s Uniform Crime Reports archives to make this chart. Banning “assault rifles” is not going to significantly reduce murders. If you want to fix that problem by regulating firearms, you’ll have to look at handguns.
Inspired by Mark Reid’s post illustrating the bimodal relationship between the density of guns in a population and the number of gun homicides, I’ve created a slightly different plot from the same data, designed to illustrate a slightly… muddier relationship. This is an expanded variant of homicides vs guns, all countries, but scaling firearm homicides by log(10) shows the relationships between low-homicide countries.
library(directlabels)
library(lattice)
guns <- read.table("guns/data/guns.csv", sep="\t", header=TRUE)
deaths <- read.table("guns/data/deaths.csv", sep="\t", header=TRUE)
oecd <- read.table("guns/data/oecd.csv", sep="\t", header=TRUE)
data <- merge(guns, deaths, by="Country")
data$OECD <- data$Country %in% oecd$Country
plot(
direct.label(
xyplot(Homicides ~ Guns, data,
group=Country,
main="Homicides vs. Guns",
xlab="Guns per 100 people",
ylab="Homicides vs 100k people",
scales=list(y = list(log = 10))),
"top.points"))
In the just-released riemann-java-client 0.0.6, riemann-clojure-client 0.0.6, riemann-ruby-client 0.0.8, and the upcoming riemann 0.1.4 (presently in master), Riemann will support two new types of metrics for events.
https://github.com/aphyr/riemann-java-client/blob/master/src/main/proto/riemann/proto.proto#L24
Nassim Nicholas Taleb has written a piece on futurism which is making me feel, well, contradictory. Apologies for my writing: fighting a killer headache this week.
Taleb asserts that the present has changed little from the past; that “futurists always get it wrong”, and that if we wish to envision the future we should subtract from the present things which do not belong. I believe the present is so different from the past that it would be shocking to humans from even a few centuries ago. Technology is culture, and our immersion in culture makes it quite difficult to understand just how unusual we are.
Update, 2018-09-01: Hi folks! I’ve gotten a lot of email and comments here, and I appreciate your enthusiasm, but this is a blog post about a table I built in my free time. Like I say in the post, I’m not a professional woodworker; I’m a hobbyist. I am also not a lumber mill. I’m a software engineer: I don’t have any tables or wood to sell. Try a local woodworker! If you’re in the San Francisco Bay Area, I can recommend Tree to Table. Cheers!
When Tyler and I rented this apartment together, we knew we wanted a table. Our common room has a linear kitchen at one end and the couch & coffee table at the other. Our plan (and in concordance with FARMHOUSE KITCHEN, DIFFERENT CHAIRS, and HALF-FILLED WALL) was to divide the two spaces with the dining table–and to get some extra counter and storage space. With tons of natural light, white walls, and blond flooring, we knew we wanted a solid, darker piece to balance the room–something with rough, warm materials. It also needed to be unusually high, to provide a standing work surface. After rejecting a few expensive and ill-sized pieces from craigslist, catalogs and furniture stores, we decided to build one ourselves.
From a hopelessly optimistic Socialist speech, a kernel of the Arts and Crafts movement I believe to be wholly true:
I think that to all living things there is a pleasure in the exercise of their energies, and that even beasts rejoice in being lithe and swift and strong. But a man at work, making something which he feels will exist because he is working at it and wills it, is exercising the energies of his mind and soul as well as of his body. Memory and imagination help him as he works. Not only his own thoughts, but the thoughts of the men of past ages guide his hands; and, as a part of the human race, he creates. If we work thus we shall be men, and our days will be happy and eventful.
Ready? Grab the tarball or deb from http://aphyr.github.com/riemann/
0.1.3 is a consolidation release, comprising 2812 insertions and 1425 deletions. It includes numerous bugfixes, performance improvements, features–especially integration with third-party tools–and clearer code. This release includes the work of dozens of contributors over the past few months, who pointed out bugs, cleaned up documentation, smoothed over rough spots in the codebase, and added whole new features. I can’t say thank you enough, to everyone who sent me pull requests, talked through designs, or just asked for help. You guys rock!
For the last three years Riemann (and its predecessors) has been a side project: I sketched designs, wrote code, tested features, and supported the community through nights and weekends. I was lucky to have supportive employers which allowed me to write new features for Riemann as we needed them. And yet, I’ve fallen behind.
Dozens of people have asked for sensible, achievable Riemann improvements that would help them monitor their systems, and I have a long list of my own. In the next year or two I’d like to build:
Write contention occurs when two people try to update the same piece of data at the same time.