The Technician

While having drinks with a colleague, the topic of improving execution within an organization came up. There had been some problems in the past few years, but this was an organization trying to operate in an unfamiliar way, and it was suggested that problems could’ve been avoided by having a conscientious operator. At one point in the conversation, the following was said;

The people who push the buttons that cause problems should be fired. Heads should roll.

Testing the outcome of infrastructure provisioning is an operator’s bread and butter. Good cookbook tests create predictable automation, and can be added to once problems are inevitably discovered. Serverspec is a huge part of ensuring that infrastructure being provisioned via chef is actually in the desired state.

Combining ruby’s builtin libraries to parse and manipulate configuration files with serverspec is a quick and simple win. We’re no longer bound by having to use overly complex, brittle regexes to ensure files are created correctly.

Here’s a few ways to pull in rubygems when writing serverspec;

• Simply use ruby include
• Add a Gemfile to your serverspec folder
• Use Gem::DependencyInstaller in the spec_helper.rb

Here’s an example of installing the inifile gem in your spec_helper.rb.

1
2
3
4
5
6

begin
  Gem::Specification.find_by_name('inifile')
rescue Gem::LoadError
  require 'rubygems/dependency_installer'
  Gem::DependencyInstaller.new(Gem::DependencyInstaller::DEFAULT_OPTIONS).install('inifile')
end

Here’s a few examples of using the ini gem we installed to make our tests better.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

require 'inifile'

describe 'conf file tests'
  program_conf = '/my/path/to/important/program.conf'

  it 'writes out a program.conf file' do
    expect(file program_conf).to be_file
  end

  # Regex test
  it 'has the correct maxmemory value' do
    expect(file(program_conf).content).to match/^program\.java\.maxmemory.*=.*512.*$/
  end

  # Lib test
  program_ini = IniFile.load(program_conf)
  it 'has the correct maxmemory value' do
    expect(program_ini['global']['program.java.maxmemory']).to eq(512)
  end
end

Parsing config files with similar libs that would be consuming them in production provides a lightweight, implied method of testing that those files are valid. It’s also a straightforward, programmatic approach to getting values out of configuration files. And not just scalar values, but lists and arrays.

Here’s a more in-depth example, using rspec.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

describe 'json config' do
  config_file = '/etc/default.json'
  conf = JSON.parse(File.read(config_file))

  it 'does *not* have the unkey key' do
    expect(conf).to_not have_key('unkey')
  end

  it 'sets foo to bar' do
    expect(conf['foo']).to eq('bar')
  end

  expected_items = [
    '127.0.0.128',
    '127.0.0.126',
    '127.0.0.127'
  ]

  describe conf['items'] do
    it { should match_array(expected_items) }
  end
end

Normally, I’m a big fan of music without lyrics or much adornment for coding, but for the past few months, there have been a few albums in my rotation that are pretty heavy on the lyrics and complications. This edition has four guitar heavy albums, and three with way more singing than usual.

Antemasque, Antemasque

Antemasque is a newish project from Omar Rodriguez-Lopez, from Mars Volta and At the Drive-In fame. While Mars Volta have long been a favorite for coding, thanks to the free form jazz and overall chaotic energy, Antemasque is significantly more straight ahead progressive rock. Busy guitars, with only a light coat of fuzz, push forward and provide a sonic landscape you can lose yourself in. Cedric Bixler-Zavala’s vocals are elegantly mixed in, riding right alongside the guitars. Most of album has a quick, steady drive, with only a quick slowdown for “Drown All Your Witches” and “Providence”.

Monuments, Ume

I found this album through Crave Online’s Best Albums of 2014. I’m a huge sucker for a band with a big sound and a strong female front, having spent a lot of time listening to the Joy Formidable after seeing them at SXSW years ago. Ume (pronounced ooh-may) is usually a bit more of what I’m looking for, tempowise, in coding music. Most songs are a laid back andante, moved along by thrumming bass. What I like most is Lauren Larson’s silky, gently echoing voice over shiny, just shy of screeching guitar licks.

Gangs, And So I Watch You from Afar

As creepy as this band’s name is, they are anything but. Their more recent album, All Hail Bright Futures, was contagiously happy and comforting, but that’s for another post. Gangs, on the other hand, is certainly non-threatening, but has a big enough punch to kick your ass even through Apple earbuds. Both albums were instrumental, and as such have been a big part of my Spotify rotation. The standout track for me is “Search:Party:Animal”, with a desperate, machine guitar riff driving away, punctuated by mountainous bass guitar hits. Interwoven throughout are quiet, but perturbed bass interludes that eventually culminate in the band coming together for builds that drop into high jitter. The whole thing ends in a midrange modulation turned into a samba-esque power chord riff with an abrupt but tasty ending. The rest of the album presents the same mountainous walls of guitar distortion.

Gemini, Her Majesty, RX Bandits

The RX Bandits were a favorite of mine a few years ago, and Gemini, Her Majesty is their first record in five years. Again, big thanks to Crave Online for uncovering this one. What I like here is the relaxed, smooth, yet exuberant crooning over bright SoCal guitar riffs. There’s lots of forward movement, as most of the album has a quick pace, driving at times, but powerful riffs where the whole band hits together.

Your time is extremely valuable, and execution of your product is everything. When it comes to building out key pieces of infrastructure, the choice of whether to build it, buy it, or use open source is very personal, and depends on what the business really needs.

You do things yourself when you need fine grained control of a resource, or require a deep understanding of a mission critical function. Can anyone do it better than you can? Even if they can, do you have enough control in the direction to get what you need? Does the time it takes to learn and execute in a given domain warrant the value it provides? Or could that time be spent on something else that would move the needle?

Dead giveaways are things the user sees. Design and product development can’t be outsourced, nor can the actual execution of your product. Those stay in house, no matter what. Your ability to find your voice and interact with your customers is the thing that makes your product unique. If you cannot execute on your own product, it’s a strong indication you’re not doing the right thing.

However, depending on your needs, there gets to be a long list of things that aren’t your product, but are still critical to running your business. Campaign and transactional email are great examples. Executing on mass email is extremely difficult because of defenses put in place by major ISPs. You need an IP with a good reputation, reverse DNS, a host that will allow massive amount of email to leave it’s datacenter, and the list goes on and on.

There are also implicit supporting requirements of running a service that you need to have but shouldn’t ever try to build. Project tracking, monitoring, graphing, and alerting are all examples of components that have executed extremely well by others, and can simply be turned on or deployed with minimal effort on the part of your team.

In the end, what’s critical is spending your energy on the things that actually create impact. Knowing the intersection of tradeoffs and priorities is what will keep your team on the path that gets the job done.

A few weeks ago, I was stuck in an airport for a couple hours. I was flying to San Francisco, and upon arriving to JFK, was informed my flight would be delayed several hours due to a ground delay program. A ground delay program is put in place when a particular airport’s ability to keep up with traffic is diminished. If your destination airport has a program in place, your takeoff will be delayed until the destination recovers.

A ground delay program is the operations equivalent of shedding load. ATC essentially stops allowing inbound flights to take off to allow for the extra time and care it takes for en-route flights to land safely. This is very similar to Netflix’s implementation of the CircuitBreaker pattern, as it allows the resource having trouble to recover while keeping flights en route at a minimum.

While I was stuck, I found there’s a couple really neat tools that can tell you a little about your chances of getting there on time.

FAA

The FAA is kind enough to supply information on delays here. However, I found the information to be presented here a bit disingenuous. The map continued to show my destination airport as green, although flights were actually being held at departure points.

FlightStats.com

They have a delay index where every destination airport is given a ranking of 0 through 5, 0 being totally on time, and 5 meaning go back to the airport bar and grab another beer. They also very helpfully track whether that index is trending up or down. These relate directly to how long a Ground Delay Program has been running. The longer that ground delay program runs, they more likely that index is going to going to trend up. They also provide a listing of flights to your destination airport that you can use to benchmark how delayed other flights and airlines are.

The past year has been an extremely busy one. There have been a number of challenges placed in front of me that were somewhat all-consuming. While there’s been plenty of introspection, much of it has been exceedingly private. My goal for the end of the year is to consider what I’ve learned in the past 9 or so months, and share it publicly. Wish me luck.

This Christmas, Jessica and I took a trip to Napa Valley. In the past year, I’ve taken quite a few trips out to California, but most of them have been helicopter trips for work. I’ve flown in, done what I needed to do, and flown out, as quickly as possible. These trips have been grueling, as they were designed to get me home quickly, and never allowed any time for relaxation. It seemed important to take a trip focused on getting to know an area.

We decided that we were going to spend a few days in Napa, so naturally, a B&B was a great choice. We wanted to have breakfast provided, a bunch of other folks to chat with about life, and a warm, comfy place to call home for a couple days. We found the Inn on Randolph via the Googlez, and were impressed by the comfortable looking rooms, so we booked. They offered a wine tour through Platypus Tours so we booked it. It seemed like a good idea for 2 folks who haven’t been behind the wheel in years to not be behind the wheel and drinking. That’s about all the planning we did.

Upon arriving, we were greeted warmly, given freshly baked cookies and the lay of the land. Both Karen and Stacey were immensely knowledgeable about navigating Napa, and were able to recommend great places based on how we felt. What’s more, it seems the Inn on Randolph has taken advantage of a great network of wine makers, restaurateurs, and tasting rooms to provide a great experience. Each morning at breakfast, Karen or Stacey would ask if we had plans, and if not, could they help. On their recommendation, they booked us into amazing experiences. For wineries they couldn’t book us into, they provided tasting cards. Much of the value of staying at the Inn is the advice and access (read: free tastings) they provide. However, expect that to evaporate into wine purchases, as the recommendations will quickly turn into opportunities to buy very unique and delicious wines.

The Inn itself is gorgeous. It has a warm, comfortable palette of dark wood, grays, and cremes decorated with Victorian furniture. The Inn has also paid close attention to creature comforts that make for a truly relaxing stay away from home. The bathroom floors were heated, which makes the Inn the most luxuriant place I have ever stayed at. The beds were the kind that hug you and don’t let go, with heavy comforters that make it difficult to leave. (The only way I was able to get up was knowing my feet wouldn’t freeze on cold floors.)

All in all, this was an incredibly warm, comfortable way to spend a few days in wine country. The Inn aims to send folks to places that will educate and treat them well, and to provide a delightful place to roost at once they’re done.

Since moving to New York, and discovering that there are more restaurants per square mile than any other place in the world, I’ve started eating out a lot. A probably unhealthy amount. But that’s OK, because everyone’s got to eat, right? And after eating at lots of places, I’ve discovered that there are certain gold standard dishes that most places are beholden to. The two that I’ve focused on are Eggs Benedict, and the humble hamburger. These are two meals that are basic, but can be riffed on infinitely.

Eggs Benedict is arguably the more complex of the two, given how hard it can be to poach a goddamn egg properly. Also, since it’s a breakfast, a bad eggs benny can put a serious damper on your day. However, despite the whole poaching challenge, I’ve rarely seen it mangled. There’s even room for quite a bit of variance. A bit of apple cider vinegar in the water can impart a tangy flavor. A few seconds can make the difference between a completely liquid yolk and a more viscous one. Then we get to the bread (soda bread being a unique standout, at Wilfie and Nell. This is the foundation of the dish, so it can really make or break it. For example, an overdone, rubbery English muffin can be so challenging to even the sharpest steak knife that you wind up shredding the whole meal. Breakfast should never be a workout. Hollandaise sauce is yet another canvas which can be painted on in endless ways. It accepts most seasonings surprisingly well. Dill is my favorite so far. Then you have the pig portion of the meal. Ham steak, streaky bacon, it’s all fair game.

The you have the burger, the old American stalwart. Again, super hard to screw up, but even harder to stand out. You also don’t have to wait in line at Umami Burger to get a good one. The blend of meat that goes into the patty (LaFrieda is king here), the cheese, bun all have a universe of possibilities. In my opinion, the more fat you start with in your meat, the better. Any burger that has short rib within 10 feet of it’s name is almost guaranteed to have a great flavor and texture. As pricy as it was, the $25 Black Label burger at Minetta Tavern was really something special. The patty there was made of prime dry-aged neef cuts. The choice of caramelized onions was awesome, as was skipping the cheese, as the patty stood completely on its own. Cave aged cheddar, which Peels employs, has made for a notable meal. And of course, the bun is there to keep your fingers (relatively) clean, or just fall apart. It doesn’t even have to be a traditional bun. Whitman’s makes a patty melt that comes between two slices of a Blue Ribbon Pullman loaf.

The best thing about these two dishes is that they’ll never get old. As long as restaurants keep experiments, coming and going, there’s always going to be an awesome variation!

Robert Frost extolled the path not taken in his famous poem. It was a tribute to bucking the trend, and finding a place that was your own. In the go go go world of the tech industry, there are fewer paths untrodden, and many of them look wrong. And yet there are just as many reasons to go along those paths. Those reasons are specific to each context, and may look sunnier or darker depending on what lays along that path.

After seeing many of these paths, and knowing that each one looks different to everyone, the only way to truly determine the wrong path is to wholeheartedly walk down a path as if it were the right one. My favorite learning experiences have been when I pursued paths that seemed right, but were not. When determining the right path to take, sometimes the best thing to do is pick a path, walk down it, and see if you get where you need to go.

Here’s to walking down each path as if it were right.

The internet is hostile. Not because of trolls or flame wars or your opinion on gay marriage, but because people want to destroy you. For absolutely no fucking reason. DDoS attacks are the scariest thing in the world to someone running a website.

The alerts will trickle in at first. It’ll just be a web sever or two that’s squawking. Then more. Then external monitoring will go off. Pingdom will mark you as down, a painful insult to your hard work, and numerous nines. Then all of the web servers will alert as down. And those alerts will keep coming. For a large infrastructure, potentially hundreds. You’ll have to quit email, or turn off notifications, or the cacophony of dings and vibrations will rattle around your brain and wrestle away whatever modicum of clarity you may have. SSH hangs, pings fail, your jump server gets squirrelly, and panic mounts. Tell your boss to get on chat. Don’t email, text, or call, because those channels will be fucked, occupied by automated alerts, hosting providers, vendors, and other team members.