Miscellaneous Open Source Contributions

I'll take Mercurial over git any day for all the reasons obvious to anyone who's really used both of them, but geeyah github sure makes contributing to projects easy. At work we had a ten minute MongoDB upgrade downtime turn into two hours, and when we finally figured out what deprecated option was causing the daemon launch to abort, rather than grouse about it on Twitter (okay, I did that too) I was able to submit a one line patch without even cloning down the repository that got merged in.

On the more-substantial side I fixed some crash bugs in dircproxy. It had been running rock solid for me for a few years, but a recent libc upgrade that added some memory checking had it crashing a few times a day. Now (with the help of Nick Wormley) I was able to fix some (rather egregious) memory gaffs. I guess this is the oft trumpeted advantage of open source software in the first place -- I had software I counted on that stopped working and I was able to fix. Really though it was just fun to fire up gdb for the first time in ages.

Finally, I was able to take some hours at work and contribute a cookbook for chef to add the New Relic monitoring agent to our many ec2 instances. It may never see a single download, but it's nice to know that if someone wants to use chef to add their systems to the New Relic monitoring display they don't have to start from scratch.

I've been living in a largely open source computing environment for fifteen years, but the barrier to entry as minor contributor has never been so low.

Ancient Content Warnings

I just rebuilt the ry4an.org server, and as part of the migration I realized a still had a lot of very old, almost embarrassing content online. I took the broken or not-conceivably interesting stuff off-line and am serving up 410 GONE responses for it.

There exists, however, a broad swath of stuff that's not yet entirely useless, but is more than ten years old and not stuff I would likely post today. For all of these pages I've left the content up, but you first have to click through a modal dialog warning you you're looking at very old stuff I don't necessarily endorse. That pop up looks like this:

Road Rage Races with an Ancient Content warning

An example can be found here: https://ry4an.org/rrr/ . (Though, if anyone still wants to race from point to point in the twin cities during the worst of rush hour I still think it's an awesome idea.)

Being the sort of person that I am I automated the process of adding those warnings to anything that hasn't been modified in at least 10 years. So, if you got an ancient content warning when viewing this page: Hello 2021!

If you follow a link or bookmark to ry4an.org and you get a 404 Not Found, let me know. Everything should either still be there or should give a 410 GONE so you know it's not there on purpose.

Asynchronous Python Logging

The Python logging module has some nice built-in LogHandlers that do network IO, but I couldn't square with having HTTP POSTs and SMTP sends in web response threads. I didn't find an asynchronous logging wrapper, so I wrote a decorator of sorts using the really nifty monkey patching availble in python:

def patchAsyncEmit(handler):
    base_emit = handler.emit
    queue = Queue.Queue()
    def loop():
        while True:
            record = queue.get(True) # blocks
            try :
            except: # not much you can do when your logger is broken
                print sys.exc_info(
    thread = threading.Thread(target=loop)
    thread.daemon = True
    def asyncEmit(record):
    handler.emit = asyncEmit
    return handler

In a more traditional OO language I'd do that with extension or a dynamic proxy, and in Scala I'd do it as a trait, but this saved me having to write delegates for all the other methods in LogHandler.

Did I miss this in the standard logging stuff, does everyone roll their own, or is everyone else okay doing remote logging in a web thread?

Graduation Form Letter

We just passed through another graduation season, and for the second year running I was able to get by with the same stack of form letters:

Dear _______________________________,

My ( Congratulations | Condolences | ___________________ ) on your recent (
Graduation | Eagle Rank | Loss | ___________________ ).  It is with ( Great Joy
| a Heavy Heart ) that I received the news.  I'm sure it took a lot of ( Hard
Work | Cigarettes ) to make it happen.

I'm sure you'll have a ( great time | good cry ) at the ( open house | wake )
and ( regret | am glad) that I ( can | cannot ) attend.

As you move on to your next phase in life please remember:

[  ] the importance of hard work
[  ] the risks of smoking
[  ] there are other fish in the sea
[  ] don't have your mom send out your graduation invites -- you're an adult now
[  ] ________________________________________

and the value of personal correspondence.

Sincerely, your (Cousin | Scoutmaster | Parolee | _____________________),

Ry4an Brase

Enclosures (1): ( Check | Card | Gift | Best Wishes )

It's available as a Google Doc.

Scholars Walk Time Traveller

The University of Minnesota has a Scholar's Walk which celebrates great persons affiliated with the U and the awards they've won. One display labeled "Historical Giants" remains without any names. Since the U can't reasonably be anticipating any new history, I imagine that four years after installation there's still a committee somewhere arguing about which department gets more names. Not content to wait for committee I decided to add a historical giant of my own -- a time traveller.

Modified Scholar's walk display

The displays are stone boxes with two panes of glass. The outermost pane of glass extends a quarter inch on steel pegs and shows the University branding and the category information. The innermost pane is recessed three inches and contains the names of the honorees. I figured that the outermost glass would provide the necessary gloss, and that any way I could get names behind it and at the right depth would look okay.

I had vinyl lettering made with the wording I wanted and applied it to some laboriously-cut thin lexan. In tests I could I could bend the lexan ninety degrees with just a four inch radius. That flexibility was enough that I could slide the insert in the bottom of the display (the top was sealed against weather).

I showed up early one morning, slipped the insert into place between the two pieces of glass, and was very pleased with the result. The white lettering looked sufficiently etched once behind the shiny outer glass, the fonts and sizes matched nearby displays, and the borders of the lexan insert were nearly invisible. The text is easier to read in the big photo links below, but it lists a physicist with a dis-joint lifespan lauded "for her uniquely important contributions to the understanding of time travel".

Sadly, the insert was removed a few months later, and before someone who can actually take a decent picture got to it. Removing it probably meant disassembling the box to some extent as the very springy plastic insert wouldn't adopt the bend required to make it around the corner without significant coercion not applicable from out side the box.

Here are the bigger pics:

Homemade Tonic

I just made my third batch of tonic water from Mark Sexauer's Recipe:

  • 4 cups water
  • 4 cups sugar
  • 1/4 cup cinchona bark
  • 1/4 cup citric acid
  • Zest and juice of 1 lime
  • Zest and juice of 1 lemon
  • Zest and juice of 1 orange
  • 1 teaspoon coriander seeds
  • 1 teaspoon dried bitter orange peel
  • 10 dashes bitters
  • 1 hand crushed juniper berry (I used two)

The flavor is excellent, but the process is terrible. Specifically, filtering the cinchona bark from the mixture after extracting the quinine (actually totaquine) from it is nearly impossible. It's so finely ground it clogs any filter, be it paper or the mesh on a french press coffeemaker, almost immediately. I've tried letting it settle and pouring off the liquid, forcing the liquid through with back pressure, and letting it drip all night -- none work well. A friend using the same recipe build a homemade vacuum extractor, but I've not yet gone that far.

This time I used multiple coffee filters and gravity, changing out the filter after each few tablespoons of water flowed through and the filter was plugged. I lost about half the liquid volume in the form of discarded, soggy filters. On previous batches I didn't take the liquid loss into account when adding sugar, yielding tonic water that was too sweet -- my primary complaint with store bought tonic --, but this time I halved the sugar as well and got a nice tart result. The fresh citrus and two juniper berries I use make it a perfect match for a nice gin. I'm able to carbonate the mixture with my own carbonation rig.

For the next batch I'm going to see if I can buy some quinine from the gray market online pharmacies. A single 300mg tablet, used to fight malaria or alleviate leg cramps, will make three strong liters and I'll be able to skip the filtering process entirely. The other benefit to using quinine instead of totaquine is aesthetic. The cinchona bark leaves my homemade tonic syrup brown and the water an unpleasant yellow color as you can see below.

Tonic Water and Syrup

A Few Quick EC2 Security Group Migration Tools

Like half the internet I'm working on duplicating a setup from one Amazon EC2 availability zone to another. I couldn't find a quick way to do that if my stuff wasn't already described in Cloud Formation templates, so I put together a script that queries the security groups using ec2-describe-group and produces a shell script that re-creates them in a different region.

If all your ec2 command line tools and environment variables are set you can mirror us-east-1 to us-west-1 using:

ec2-describe-group | ./create-firewall-script.pl > create-firewall.sh

With non-demo security group data I ran into some group-to-group access grants whose purpose wasn't immediately obvious, so I put together a second script using graphviz to show the ALLOWs. A directed edge can be read as "can access".

Security group access grants

That script can also be invoked as:

ec2-describe-group | ./visualize-security-groups.pl > groups.png

The labels on the edges can be made more detailed, but having each of tcp, udp, and icmp shown started to get excessive.

Both scripts and sample input and output are in the provided tarball.

Using the Khan Academy to Vet Online Panhandlers

I'm a huge fan of the Khan Academy (and if you haven't yet watched Salman's presentation at TED2011 you should go do that). I'm involved (slightly) in an effort to bring Khan Academy instruction to a local school district and have a standing offer to "coach" participants. Today, though, I found a use for the Khan Academy site probably doesn't endorse: dealing with online panhandlers.

Before today I'd never seen an IRC panhandler, email sure, but never IRC. This morning, though, someone wandered into #mercurial asking for money to renew a domain. Most people ignored the request since it's grossly inappropriate in a software development work space, but I offered the kid the money if he'd do algebra problems on the Khan Academy site.

We dropped into a private chat and with a mixture of whining and negotiation struck upon a deal wherein he'd do some math and I'd paypal him a few bucks. The whole exchange was silly and I got nothing out of it except a blog entry, but the math did serve some small purpose: it made sure the exchange took time and attention for long enough to not be something that the kid could be simultaneously working in multiple other channels at the same time. Also who doesn't need to known mean, median, and mode.

The Khan Academy coach interface provided me real-time feedback on the problems, including seconds spent on each, what they were, and how he answered. He might have been plugging them into Wolfram Alpha or asking in another IRC channel, but I didn't get that sense, and the time consumed would be the same either way.

Khan Academy coach interface

Surprisingly, further chat proved out the details in the initial pitch. The domain is now registered through 2014 and hosts a nice blog. Whois info, blog identity, and paypal address all align with a Florida teen. Not that I would have cared if it was a gold farmer in China -- though their thirteen year olds can do algebra.

Other kids: For the record, I'm not interesting in giving you money for math, or anything else, but if you add ry4an at brase.com as a coach on Khan Academy I'm happy to help with math.

Below are some of the choice parts from the full chat transcript:


reStructuredText Resume

I've had a resume in active maintenance since the mid 90s, and it's gone through many iterations. I started with a Word document (I didn't know any better). In the late 90s I moved to parallel Word, text, and HTML versions, all maintained separately, which drifted out of sync horribly. In 2010 I redid it in Google Docs using a template I found whose HTML hinted at a previous life in Word for OS X. That template had all sorts of class and style stuff in it that Google Docs couldn't actually edit/create, so I was back to hand-editing HTML and then using Google Docs to create a PDF version. I still had to keep the text version current separately, but at least I'd decided I didn't want any job that wanted a Word version.

When I decided to finally add my current job to my resume, even months after starting, I went for another overhaul. The goal was to have a single input file whose output provided HTML, text, and PDF representations. As I saw it that made the options: LaTeX, reStructuredText, or HTML.

I started down the road with LaTeX, and found some great templates, articles, and prior examples, but it felt like I was fighting with the tool to get acceptable output, and nothing was coming together on the plain text renderer front.

Next I turned to reStructuredText, and found it yielded a workable system. I started with Guillaume ChéreAu's blog post and template and used the regular docutils tool rst2html to generate the HTML versions. The normal route for turning reStructuredText into PDF using doctools passes through LaTeX, but I didn't want to go that route, so I used rst2pdf, which gets there directly. I counted the reStructuredText version as close-enough to text for that format.

Since now I was dealing entirely with a source file that compiled to generated outputs it only made sense to use a Makefile and keep everything in a Mercurial repository. That gives me the ability to easily track changes and to merge across multiple versions (different objectives) should the need arise. With the Makefile and Mercurial in place I was able to add an automated version string/link to the resume so I can tell from a print out which version someone lis looking at. Since I've always used a source control repository for the HTML version it's possible to compare revisions back to 2001, which get pretty silly.

I'm also proud to say that the URL for my resume hasn't changed since 1996, so any printed version ever includes a link to the most current one. Here are links to each of those formats: HTML, PDF, text, and repository, where the text version is the one from which the others are created.

Trying Hirelite

Tonight I tried Hirelite, and I really think they're on to something. They arrange online software developer interview speed-dating-style events where N companies and N software developers each sit in in front of their own webcams and are connected together for live video chats five minutes at a time. It's like chatroulette with more hiring talk and less genitalia.

They pre-screen the developers to make sure they're at least able to solve a simple programming problem and screen the companies by charging them a little money. Each session has a theme, like "backend developers", and a geographic region associated with it. After each five minute interview -- there's a countdown timer at the top -- each party indicates with a click whether or not they'd like to talk further. Mutual matches get one another's contact information immediately afterward.


I've only tried Hirelite once, which was only nine interviews, but from my limited experience here are some of my observations:

  • No one is ready for an interview-like talk that lasts only five minutes. One party or another launches into a sales pitch and uses up two or three minutes right off the bat, leaving little time for a more bidirectional exchange.
  • When it's clearly a bad skills/interests match, five minutes is still easy to fill. If two people can't find something, anything to talk about for five minutes something is wrong with one or both of them.
  • With only five minutes it's less like an interview, or even a phone pre-screen, and more like a hand-delivery of a resume.
  • Race and gender aren't always knowable from a resume, but they immediately are when a video chat happens this early in the process. Conceivably there could be some EoE considerations there, be they concious or not.
  • The novelty of what everyone is doing was enough to keep things light. One of my interviews started with an interviewer holding up a sheet of paper that said "USE TEXT CHAT" where he explained his mic wasn't working. How everyone dealt with the technical glitches is probably as good an indicator of personality as anything else -- I did an entire exchange using text-chat and pantomime.

Tips for Participants

Again, I've only done this once, but here are some tips I intend to employ if I try this again:

  • Make sure to wear headphones -- without them you'll echo.
  • If you're using a laptop have an external keyboard plugged in -- you don't want to have to hunch forward to use the one on your laptop if it comes to text chat.
  • Interviewees: Make sure to include a resume link in your pre-interview profile. Most interviewers had looked at mine, and the rest did so during the chat.
  • Check your lighting before starting. I looked like I was sitting in a cave.

Technical Issues

I'm sufficiently enamored with the system that the technical glitches were more funny than they were frustrating, but I could easily see someone else feeling otherwise. Here's what I encountered.

Before the session began I followed Hirelite's advice to test my "video and audio" with their test interview link. My camera worked fine -- I could see myself -- but I couldn't hear myself on audio. I spoke to Hirelite about that, and it turns out you're not supposed to hear yourself during the test -- there is no test for audio.

I tried both Google Chrome on Linux and Firefox on Linux with the test, and both worked. It wasn't until my audio stopped working well during an interview that I realized the text chat wasn't working in Chrome -- the text entry box was cutoff on my small netbook screen. A quick switch to Firefox got that going.

The most persistent technical problem I had was the lagging of my outbound audio. I could generally see and hear the interviewer with little lag and in synch, and my interviewer and I could both see my video unlagged, but my audio was delayed by about 40 seconds starting about the sixth interview. This made speaking all but impossible.

My internet connection speed just tested out at 14Mbps downstream and 5Mbps upstream which is more than sufficient for small video and low bitrate audio, so I don't think the problems were with my bandwidth. What's more, since Hirelite is built in Flash (who does that this decade?!) likely using RTFMP (I'm kicking myself for not running Wireshark during the session) audio and video are multiplexed in the same packet stream -- so any de-synch-ing of the two is almost certainly a software problem not a network problem.

When things were glitching I was often able to improve them temporarily by reloading the page. This kills off the Flash component and reloads it, which brought the lag down for awhile. The Hirelite system was able to re-connect me to my in-progress session with nary a hiccup, which is a nice touch. I was afraid it would automatically move me to the next interviewer.

All in all a great idea. StackExchange should buy Hirelite using some the 12 million they just raised, hopefully some of the money from their snack room.