Previously on Jepsen, we introduced the problem of network partitions. Here, we demonstrate that a few transactions which “fail” during the start of a partition may have actually succeeded.

Postgresql is a terrific open-source relational database. It offers a variety of consistency guarantees, from read uncommitted to serializable. Because Postgres only accepts writes on a single primary node, we think of it as a CP system in the sense of the CAP theorem. If a partition occurs and you can’t talk to the server, the system is unavailable. Because transactions are ACID, we’re always consistent.


Continue reading (1083 words)

This article is part of Jepsen, a series on network partitions. We’re going to learn about distributed consensus, discuss the CAP theorem’s implications, and demonstrate how different databases behave under partition.


Continue reading (1384 words)

Riemann 0.2.0 is ready. There’s so much left that I want to build, but this release includes a ton of changes that should improve usability for everyone, and I’m excited to announce its release.

Version 0.2.0 is a fairly major improvement in Riemann’s performance and capabilities. Many things have been solidified, expanded, or tuned, and there are a few completely new ideas as well. There are a few minor API changes, mostly to internal structure–but a few streams are involved as well. Most functions will continue to work normally, but log a deprecation notice when used.

I dedicated the past six months to working on Riemann full-time. I was fortunate to receive individual donations as well as formal contracts with Blue Mountain Capital, SevenScale, and Iovation during that time. That money gave me months of runway to help make these improvements–but even more valuable was the feedback I received from production users, big and small. I’ve used your complaints, frustrations, and ideas to plan Riemann’s roadmap, and I hope this release reflects that.

Continue reading (941 words)

The Netty redesign of riemann-java-client made it possible to expose an end-to-end asynchronous API for writes, which has a dramatic improvement on messages with a small number of events. By introducing a small queue of pipelined write promises, riemann-clojure-client can now push 65K events per second, as individual messages, over a single TCP socket. Works out to about 120 mbps of sustained traffic.


I’m really happy about the bulk throughput too: three threads using a single socket, sending messages of 100 events each, can push around 185-200K events/sec, at over 200 mbps. That throughput took 10 sockets and hundreds of threads to achieve in earlier tests.

Continue reading (219 words)

In the previous post, I described an approximation of Heroku’s Bamboo routing stack, based on their blog posts. Hacker News, as usual, is outraged that the difficulty of building fast, reliable distributed systems could prevent Heroku from building a magically optimal architecture. Coda Hale quips:

Really enjoying @RapGenius’s latest mix tape, “I Have No Idea How Distributed Systems Work”.

Coda understands the implications of the CAP theorem. This job is too big for one computer–any routing system we design must be distributed. Distribution increases the probability of a failure, both in nodes and in the network itself. These failures are usually partial, and often take the form of degradation rather than the system failing as a whole. Two nodes may be unable to communicate with each other, though a client can see both. Nodes can lie to each other. Time can flow backwards.

Continue reading (2140 words)

For more on Timelike and routing simulation, check out part 2 of this article: everything fails all the time. There’s also more discussion on Reddit.

RapGenius is upset about Heroku’s routing infrastructure. RapGenius, like many web sites, uses Rails, and Rails is notoriously difficult to operate in a multithreaded environment. Heroku operates at large scale, and made engineering tradeoffs which gave rise to high latencies–latencies with adverse effects on customers. I’d like to explore why Heroku’s Bamboo architecture behaves this way, and help readers reason about their own network infrastructure.

To start off with, here’s a Rails server. Since we’re going to be discussing complex chains of network software, I’ll write it down as an s-expression:

Continue reading (1708 words)

I’m not a big fan of legal documents. I just don’t have the resources or ability to reasonably defend myself from a lawsuit; retaining a lawyer for a dozen hours would literally bankrupt me. Even if I were able to defend myself against legal challenge, standard contracts for software consulting are absurd. Here’s a section I encounter frequently:

Ownership of Work Product. All Work Product (as defined below) and benefits thereof shall immediately and automatically be the sole and absolute property of Company, and Company shall own all Work Product developed pursuant to this Agreement.

“Work Product” means each invention, modification, discovery, design, development, improvement, process, software program, work of authorship, documentation, formula, data, technique, know-how, secret or intellectual property right whatsoever or any interest therein (whether or not patentable or registrable under copyright or similar statutes or subject to analogous protection) that is made, conceived, discovered, or reduced to practice by Contractor (either alone or with others) and that (i) relates to Company’s business or any customer of or supplier to Company or any of the products or services being developed, manufactured or sold by Company or which may be used in relation therewith, (ii) results from the services performed by Contractor for Company or (iii) results from the use of premises or personal property (whether tangible or intangible) owned, leased or contracted for by Company.

These paragraphs essentially state that any original thoughts I have during the course of the contract are the company’s property. If the ideas are defensible under an IP law, I could be sued for using them in another context later. One must constantly weigh the risk of thinking under such a contract. “If I consider this idea now, I run the risk of inventing something important which I can never use again.”

Continue reading (1568 words)

Michael Robertson writes:

@Jason @MicahSingleton Biz can pursue profits or racism, but not both. Tech industry is a meritocracy as is all industries in a free market.

and Jason Calacanis adds:

Continue reading (1060 words)

I have it pretty good, in America. I’m White, male, young. Grew up with books. With enough food on the table during critical phases of brain development. In a neighborhood composed of people who looked and spoke like me, a neighborhood with a creek, and trees, and street hockey, somewhere safe. Through deterministic happenstance–a confluence of genetics and education and economics and municipal investment in public education and intellectually challenging parents and the right teachers at pivotal moments–I’m good at thinking about a class of problem which too few people are working on, and present market dynamics allow me to do what I love for far more money than I need.

People grant me the authority to speak as is expected of males, with the lack of recognition of my skin color that comes for people of northern European origin, and for my youth I am forgiven all manner of brash and disrespectful rejoinders. I am significantly more likely to be a victim of a murder, and feel constant pressure to be resolute, correct, gruff. I have never worried for my physical safety in the presence of male companions, and think nothing of walking alone at night. As a motorcyclist and as an engineer I am never the odd one out. I can wear comfortable clothes at formal gatherings. I can enter any building freely, and when boarding a bus, folks never rustle and stare at the delay. I feel tremendously self-conscious when surrounded by people of color. My coworkers never comment about how pretty I am. I am never expected to speak for all young, White males.

I will never have the experience of being a woman; to keep it together when my male coworkers take credit for my ideas and I’m still making $20,000 less than the junior devs fresh out of college. To move from Pakistan to the rural Midwest, to a new culture and bureaucracy, and struggling to learn math my classmates can barely cope with–in a language not my own. To be told all my life that I was White, because the family that adopted me was so much darker, and I was told I had beautiful light skin and to keep from getting tan, and when I moved to New York they called me Black, but I don’t know how to be Black and no one will teach me. To be the only kid from the Rez who went to college, and to have pleaded with admissions to let my dad off the hook for when he wouldn’t lift a fucking drunk-ass finger to fill out the FAFSA, and to remember my grandmother’s strength and intellect and passion: my inspiration always.

Continue reading (2126 words)

tl;dr Riemann is a monitoring system, so it emphasizes liveness over safety.

Riemann is aimed at high-throughput (millions of events/sec/node), partial-harvest event processing, where it is acceptable to trade completeness for throughput at low latencies. For instance, it’s probably fine to drop half of your request latency events on the floor, if you’re calculating a lossy histogram with sampling anyway. It’s also typically acceptable to have nondeterministic behavior with respect to time windows: if one node’s clock is skewed, it’s better to process it “soonish” rather than waiting an unbounded amount of time for it to check in.

There is no synchronization or relationship between events. Events are immutable and have a total order, even though a given server or client may only have a fraction of the relevant events for a system. The events are, in a sense, the transaction log–except that the semantics of those transactions depend on the stream configuration.

Continue reading (1143 words)

Mass distribution of learning material has been around for a few centuries and has yet to replace the process of guided learning. While it’s possible to amass facts and skills from reading and listening, it’s much more difficult to produce complex works of value without feedback on the process.

Doing mathematics isn’t just applying rules and techniques. It’s about knowing how to reason, and writing a proof in a way which communicates your reasoning clearly to others. You can get started by following along with proofs from a lecture, but in order to really ingrain the techniques in your brain, you have to write proofs of things you’ve never encountered before. Someone has to read those proofs, and give feedback on where your reasoning was unclear, incomplete, or flawed. They can suggest a different notation, or a shorter path to the same solution. Good teachers will leave notes: “this is a cool idea you’ve developed here, and it points towards this area of complex analysis we haven’t talked about yet.”

In psychology it’s not enough to memorize a summary text and a smattering of papers. You need to be asked questions. “There’s a critical flaw in this paper’s sampling methodology. Can you find it? How would you improve it?” “What kind of systematic bias can we expect in these results?” If nobody asks those questions, and helps you hone in on the answers, you’ll miss out on half the text. You’ll be unprepared to evaluate the quality of others research–or to design experiments of your own.

Continue reading (276 words)

We got to talking about space warfare last night, and I realized something pretty weird: FTL drives effect massive shifts in velocity.

Almost every FTL spacecraft, in fiction, is capable of moving between planets in different star systems. The ship starts out roughly stationary relative to planet A, and winds up roughly stationary relative to planet B. How fast are A and B moving compared to one another? How fast do stars move?

Proxima Centauri has a radial velocity (relative to the solar system’s center-of-mass) of -21.7 +/- 1.8 km/s. Its proper motion vector is -3.77530 arcsec/year in right ascension, and 0.76933 arcsec/year in declination. At 4.243 light-years away, its proper motion relative to sol is 23.777 km/s. Its total relative velocity to sol is somewhere around 32.19 km/s, which is just a little faster than the velocity of the earth, rotating around the sun.

Continue reading (684 words)

Copyright © 2018 Kyle Kingsbury.
Non-commercial re-use with attribution encouraged; all other rights reserved.
Comments are the property of respective posters.