John Mullerleile, Phil Kulak, and I gave a talk tonight, entitled “Scaling at Showyou.”
I gave an overview of the Showyou architecture, including our use of Riak, Solr, and Redis; strategies for robust systems; and our comprehensive monitoring system. You may want to check out:
AWS::S3 is not threadsafe. Hell, it’s not even reusable; most methods go through a class constant. To use it in threaded code, it’s necessary to isolate S3 operations in memory. Fork to the rescue!
def s3(key, data, bucket, opts)
begin
fork_to do
AWS::S3::Base.establish_connection!(
:access_key_id => KEY,
:secret_access_key => SECRET
)
AWS::S3::S3Object.store key, data, bucket, opts
end
rescue Timeout::Error
raise SubprocessTimedOut
end
end
def fork_to(timeout = 4)
r, w, pid = nil, nil, nil
begin
# Open pipe
r, w = IO.pipe
# Start subprocess
pid = fork do
# Child
begin
r.close
val = begin
Timeout.timeout(timeout) do
# Run block
yield
end
rescue Exception => e
e
end
w.write Marshal.dump val
w.close
ensure
# YOU SHALL NOT PASS
# Skip at_exit handlers.
exit!
end
end
# Parent
w.close
Timeout.timeout(timeout) do
# Read value from pipe
begin
val = Marshal.load r.read
rescue ArgumentError => e
# Marshal data too short
# Subprocess likely exited without writing.
raise Timeout::Error
end
# Return or raise value from subprocess.
case val
when Exception
raise val
else
return val
end
end
ensure
if pid
Process.kill "TERM", pid rescue nil
Process.kill "KILL", pid rescue nil
Process.waitpid pid rescue nil
end
r.close rescue nil
w.close rescue nil
end
end
There’s a lot of bookkeeping here. In a nutshell we’re forking and running a given block in a forked subprocess. The result of that operation is returned to the parent by a pipe. The rest is just timeouts and process accounting. Subprocesses have a tendency to get tied up, leaving dangling pipes or zombies floating around. I know there are weak points and race conditions here, but with robust retry code this approach is suitable for production.
In distributed systems, one frequently needs a set of n nodes to come to a consensus on a particular coordinating or master node, referred to as the leader. Leader election protocols are used to establish this. Sure, you could do the Swedish or the Silverback, but there’s a whole world of consensus algorithms out there. For instance:
Each node injects its neighbors with a total copy of its own state and identity, taking over operations on that node. Convergence is reached when all nodes are identical.
This trivial algorithm simply ensures that all nodes crash upon receiving any decapitate message from a neighbor k. That node’s responsibilities and powers are delegated to k. The last node standing wins.
Hello, law enforcement. I suspect you’re reading this because, as a TSA supervisor told me recently, “… we are interested in you”.
Yes, I asked to fly selectee–to not provide ID–at Denver International recently. Yes, I’ve done this before. Yes, there was a lot of confusion between TSA employees on whether that was legal or not–eventually M. Gatling of the DIA police told me I was required to display ID. Yes, I opted out of AIT. Yes, it did take no fewer than eight TSA officers, airline representatives, and police about 45 minutes to determine I posed no threat. Yes, I was exceedingly polite, and most of us got along quite well. Yes, I was asked all kinds of questions I was under no obligation to answer (among them my address and phone number), and no, the TSA supervisor was not very pleased that I asked whether I was legally required to respond.
“What is your contact number.”
So Technology Review published a summary of an arxiv article arguing for mass quantization in black holes. Looks like an ultraviolet catastrophe argument, which is fascinating in itself. But first, I have to address this journalistic clusterfuck:
Of course, the question of this kind of black hole production at the LHC once again raises the thorny question of whether the safety assurances we’ve been given about these experiments are valid.
No, it means the exact opposite. The article is prompted by the absence of black holes in experimental products. The implications of the model are that black holes are harder for the LHC to create.
Bob and Sarah are a physicist and chemist, respectively. They should have known better than to send us mad libs for an RSVP card.
Translation: On July 23rd, Bob and Sarah will finally be elements of the set of unordered pairs of people. Kyle is very (the official sequence ID of the “happy numbers”) for them, and wishes them an uncountable infinity of happiness and that they should be an ideal example of the wedding ring together. Therefore, 1 will, with less than 1 percent conformance with the null hypothesis, attend and look forward to seeing the undefined (as bride/groom is indivisible) in their Klein bottle dress/suit.
A fine quote from the Illustrious Miss Hirpa.
For only 650 per month, I could enjoy…
… an interest in: Group Habitation, Rocketry, Radio, Making Things, Creating Art, ReProgramming the World, Video Kaleidoscopes, Mad Science, Performing, Music or Entertaining, WareHousing, and the list goes on.
Or perhaps
If you ever need to unzip data compressed with zlib without a header (e.g. produced by Erlang’s zlib:zip), it pays to be aware that
windowBits can also be -8..-15 for raw inflate. In this case, -windowBits determines the window size. inflate() will then process raw deflate data, not looking for a zlib or gzip header, not generating a check value, and not looking for any check values for comparison at the end of the stream. (zlib.h)
Hence, you can do something like
Smear green blood
Your palms are sweaty
Scaly skin glistening
To last demands
The planet’s going under and
You’re floating overhead
(Lizard men) [x2]
Realizing that the sexism of beer advertising is just too implicit, Dos Equis helpfully spells out exactly how to treat women as subhuman.
I don’t even want to get near the colonialist thing here, either.
You’re so prolific
An impact factor well above thirty two
It makes me jealous
And in need of passing this peer review–
You know my motivations
In search of reputation
My erdos number ill acquire from you
So in this explication
Of conformal transformations
Please excuse me I don’t mean to be crude
But tonight I’m citing you
