I’m not a big fan of legal documents. I just don’t have the resources or ability to reasonably defend myself from a lawsuit; retaining a lawyer for a dozen hours would literally bankrupt me. Even if I were able to defend myself against legal challenge, standard contracts for software consulting are absurd. Here’s a section I encounter frequently:

Ownership of Work Product. All Work Product (as defined below) and benefits thereof shall immediately and automatically be the sole and absolute property of Company, and Company shall own all Work Product developed pursuant to this Agreement.

“Work Product” means each invention, modification, discovery, design, development, improvement, process, software program, work of authorship, documentation, formula, data, technique, know-how, secret or intellectual property right whatsoever or any interest therein (whether or not patentable or registrable under copyright or similar statutes or subject to analogous protection) that is made, conceived, discovered, or reduced to practice by Contractor (either alone or with others) and that (i) relates to Company’s business or any customer of or supplier to Company or any of the products or services being developed, manufactured or sold by Company or which may be used in relation therewith, (ii) results from the services performed by Contractor for Company or (iii) results from the use of premises or personal property (whether tangible or intangible) owned, leased or contracted for by Company.

These paragraphs essentially state that any original thoughts I have during the course of the contract are the company’s property. If the ideas are defensible under an IP law, I could be sued for using them in another context later. One must constantly weigh the risk of thinking under such a contract. “If I consider this idea now, I run the risk of inventing something important which I can never use again.”

Continue reading (1568 words)

Michael Robertson writes:

@Jason @MicahSingleton Biz can pursue profits or racism, but not both. Tech industry is a meritocracy as is all industries in a free market.

and Jason Calacanis adds:

Continue reading (1060 words)

I have it pretty good, in America. I’m White, male, young. Grew up with books. With enough food on the table during critical phases of brain development. In a neighborhood composed of people who looked and spoke like me, a neighborhood with a creek, and trees, and street hockey, somewhere safe. Through deterministic happenstance–a confluence of genetics and education and economics and municipal investment in public education and intellectually challenging parents and the right teachers at pivotal moments–I’m good at thinking about a class of problem which too few people are working on, and present market dynamics allow me to do what I love for far more money than I need.

People grant me the authority to speak as is expected of males, with the lack of recognition of my skin color that comes for people of northern European origin, and for my youth I am forgiven all manner of brash and disrespectful rejoinders. I am significantly more likely to be a victim of a murder, and feel constant pressure to be resolute, correct, gruff. I have never worried for my physical safety in the presence of male companions, and think nothing of walking alone at night. As a motorcyclist and as an engineer I am never the odd one out. I can wear comfortable clothes at formal gatherings. I can enter any building freely, and when boarding a bus, folks never rustle and stare at the delay. I feel tremendously self-conscious when surrounded by people of color. My coworkers never comment about how pretty I am. I am never expected to speak for all young, White males.

I will never have the experience of being a woman; to keep it together when my hormones throw me a curveball and I’m still making $20,000 less than the junior devs fresh out of college. To move from Pakistan to the rural Midwest, to a new culture and bureaucracy, and struggling to learn math my classmates can barely cope with–in a language not my own. To be told all my life that I was White, because the family that adopted me was so much darker, and I was told I had beautiful light skin and to keep from getting tan, and when I moved to New York they called me Black, but I don’t know how to be Black and no one will teach me. To be the only kid from the Rez who went to college, and to have pleaded with admissions to let my dad off the hook for when he wouldn’t lift a fucking drunk-ass finger to fill out the FAFSA, and to remember my grandmother’s strength and intellect and passion: my inspiration always.

Continue reading (2124 words)

tl;dr Riemann is a monitoring system, so it emphasizes liveness over safety.

Riemann is aimed at high-throughput (millions of events/sec/node), partial-harvest event processing, where it is acceptable to trade completeness for throughput at low latencies. For instance, it’s probably fine to drop half of your request latency events on the floor, if you’re calculating a lossy histogram with sampling anyway. It’s also typically acceptable to have nondeterministic behavior with respect to time windows: if one node’s clock is skewed, it’s better to process it “soonish” rather than waiting an unbounded amount of time for it to check in.

There is no synchronization or relationship between events. Events are immutable and have a total order, even though a given server or client may only have a fraction of the relevant events for a system. The events are, in a sense, the transaction log–except that the semantics of those transactions depend on the stream configuration.

Continue reading (1143 words)

Mass distribution of learning material has been around for a few centuries and has yet to replace the process of guided learning. While it’s possible to amass facts and skills from reading and listening, it’s much more difficult to produce complex works of value without feedback on the process.

Doing mathematics isn’t just applying rules and techniques. It’s about knowing how to reason, and writing a proof in a way which communicates your reasoning clearly to others. You can get started by following along with proofs from a lecture, but in order to really ingrain the techniques in your brain, you have to write proofs of things you’ve never encountered before. Someone has to read those proofs, and give feedback on where your reasoning was unclear, incomplete, or flawed. They can suggest a different notation, or a shorter path to the same solution. Good teachers will leave notes: “this is a cool idea you’ve developed here, and it points towards this area of complex analysis we haven’t talked about yet.”

In psychology it’s not enough to memorize a summary text and a smattering of papers. You need to be asked questions. “There’s a critical flaw in this paper’s sampling methodology. Can you find it? How would you improve it?” “What kind of systematic bias can we expect in these results?” If nobody asks those questions, and helps you hone in on the answers, you’ll miss out on half the text. You’ll be unprepared to evaluate the quality of others research–or to design experiments of your own.

Continue reading (276 words)

We got to talking about space warfare last night, and I realized something pretty weird: FTL drives effect massive shifts in velocity.

Almost every FTL spacecraft, in fiction, is capable of moving between planets in different star systems. The ship starts out roughly stationary relative to planet A, and winds up roughly stationary relative to planet B. How fast are A and B moving compared to one another? How fast do stars move?

Proxima Centauri has a radial velocity (relative to the solar system’s center-of-mass) of -21.7 +/- 1.8 km/s. Its proper motion vector is -3.77530 arcsec/year in right ascension, and 0.76933 arcsec/year in declination. At 4.243 light-years away, its proper motion relative to sol is 23.777 km/s. Its total relative velocity to sol is somewhere around 32.19 km/s, which is just a little faster than the velocity of the earth, rotating around the sun.

Continue reading (684 words)

I’ve been doing a lot of performance tuning in Riemann recently, especially in the clients–but I’d like to share a particularly spectacular improvement from yesterday.

Riemann’s TCP protocol is really simple. Send a Msg to the server, receive a response Msg. Messages might include some new events for the server, or a query; and a response might include a boolean acknowledgement or a list of events matching the query. The protocol is ordered; messages on a connection are processed in-order and responses sent in-order. Each Message is serialized using Protocol Buffers. To figure out how large each message is, you read a four-byte length header, then read length bytes, and parse that as a Msg.

Continue reading (1182 words)

I’ve had two observations floating around in my head, looking for a way to connect with each other.

Many “architecture patterns” are scar tissue around the absence of higher-level language features.

and a criterion for choosing languages and designing APIs

Continue reading (2963 words)

I’ve been putting more work into riemann-java-client recently, since it’s definitely the bottleneck in performance testing Riemann itself. The existing RiemannTcpClient and RiemannRetryingTcpClient were threadsafe, but almost fully mutexed; using one essentially serialized all threads behind the client itself. For write-heavy workloads, I wanted to do better.

There are two logical optimizations I can make, in addition to choosing careful data structures, mucking with socket options, etc. The first is to bundle multiple events into a single Message, which the API supports. However, your code may not be structured in a way to efficiently bundle events, so where higher latencies are OK, the client can maintain a buffer of outbound events and flush it regularly.

The second optimization is to take advantage of request pipelining. Riemann’s protocol is simple and synchronous: you send a Message over a TCP connection, and receive exactly one TCP message in response. The existing clients, however, forced you to wait n milliseconds for the message to cross the network, be processed by Riemann, and receive an acknowledgement. We can do better by pipelining requests: sending new requests before waiting for the previous responses, and matching up received messages with their corresponding requests later.

Continue reading (375 words)

Computer languages, like human languages, come in many forms. This post aims to give an overview of the most common programming ideas. It’s meant to be read as one is learning a particular programming language, to help understand your experience in a more general context. I’m writing for conceptual learners, who delight in the underlying structure and rules of a system.

Many of these concepts have varying (and conflicting) names. I’ve tried to include alternates wherever possible, so you can search this post when you run into an unfamiliar word.

Every program has two readers: the computer, and the human. Your job is to communicate clearly to both. Programs are a bit like poetry in that regard–there can be rules about the rhythm of words, how punctuation works, do adjectives precede nouns, and so forth.

Continue reading (3096 words)

A good friend of mine from college has started teaching himself to code. He’s hoping to find a job at a Bay Area startup, and asked for some help getting oriented. I started writing a response, and it got a little out of hand. Figure this might be of interest for somebody else on this path. :)

I want to give you a larger context around how this field works–there’s a ton of good documentation on accomplishing specifics, but it’s hard to know how it fits together, sometimes. Might be interesting for you to skim this before we meet tomorrow, so some of the concepts will be familiar.

There are two big spheres of “technical” activity, generally referred to as “development” and “operations”. Development is about writing and refining software, and operations is about publishing and running it. In general, I think development is a little more about fluid intelligence and language, and ops is more about having broad experience and integrating disparate pieces.

Continue reading (2951 words)

Schadenfreude is a benchmarking tool I'm using to improve Riemann. Here's a profile generated by the new riemann-bench, comparing a few recent releases in their single-threaded TCP server throughput. These results are dominated by loopback read latency–maxing out at about 8-9 kiloevents/sec. I'll be using schadenfreude to improve client performance in high-volume and multicore scenarios.