A while ago, I listened to a Special Sauce podcast interviewing Danny Meyer. Danny Meyer is the CEO of Union Square Hospitality Group, the organization behind Union Square Cafe, Shake Shack, and a slew of other notable restaurants. He believes there are six emotional qualities that are critical to a team member’s success. Here they are, and how they relate to being a successful tech hire.
Team members need to be hopeful. Nothing shuts down a team quicker than someone who is pessimistic about new ideas and creativity. There are tons of code problems that are solvable, and treating them as such helps create a team committed to moving forward.
Curiosity has two important ramifications; introspection into a candidates own ideas, and willingness to explore weird problems that crop up.
In tech, things change ridiculously quickly. New frameworks, methodologies and services pop up every day, and that makes every day is an opportunity to learn and apply something new. Without curiosity and introspection into the how of our work, skills stagnate.
Treating features as experiments is critical to discovering what works, and being deeply curious about interactions will help build domain knowledge for the team. When things go sideways, and they often do, in weird and unexpected ways, a sense of open curiosity is incredibly helpful in tracking down, understanding, and fixing problems.
You can’t train or incentivize someone into giving a shit about doing the work right. Many don’t realize that the success of their own work translates to success for the entire team.
Be aware of the wake you’re leaving, and care about what it does to others. Everything we say and do has consequences. Ignoring requests for help, or failing to document and discuss the work you do makes it more difficult for those you with to build on what you’ve done.
Understanding what your personal weather report is, and how it affects others. Know your weaknesses, and ask for help on decisions and work you know you are weak in. Also, understand your mood, and how it affects work. When you’re feeling in a funk, understand how to get out of it.
Does the candidate have the judgment to do the right thing, even if that thing is not in their own self interest.
The entire podcast was great, but click here to hear Meyers’s take on hiring.
With all the challenges of hiring, and vitriol around whiteboarding, focusing in on just emotional qualities is a great way for an organization to set themselves apart from the competition. Everyone needs a job, but few can have one where they have a team of great people.
]]>The viewing goes well enough, but way too quickly, and in the afternoon, so there’s no way to see all the gorgeous light it gets in the morning. Our scrappy tenant needs a place, and it’s close enough, so screw it, let’s go through with the lease signing. After the humiliating probe into credit scores, bank statements, employer letters, all eyed contemptuously by a building manager, the landlord begrudgingly accepts. Then comes the scramble for thousands of dollars in cashiers checks, for security deposit, first month rent, and brokers. This is a process designed to intimidate and humiliate. A process for which there is no landlord equivalent.
Once in an apartment, should there be any condition undisclosed during the search process, then it’s tough luck on the part of the tenant. For example, a noisy subway or bar directly underneath the apartment. Or chronic issues with vermin, heat or hot water. These are existing conditions the landlord is typically aware of, but unwilling to disclose, as landlords and management companies are incentivized to fill vacancies, not to ensure tenants enjoy their stay.
Luckily, as more data becomes available to the public, smart folks are starting to put together publicly available tools to help tenants make smart decisions about where they live. My own simple project, Block Quality, will summarize 311 complaints to paint a picture of what problems have occurred at a given address. Entire organizations like Heat Seek have formed to address the issue of tenants being denied heat and hot water. Rent Logic assigns grades to tenants, allowing tenants to know exactly what they’re getting into.
]]>Bastion and Transistor are gorgeous, ethereal iOS games put out by the venerable studio Supergiant. This Spotify playlist does a great job of gathering the two soundtracks and adding a couple remixes, for fun. While video game soundtracks should be pretty obvious background music for coding (especially the final boss themes), I’ve never found one that was as sensitive, narrative and well thought out as this. In particular, Ashley Barret’s hummed tracks are wonderful for feeling the melody while not having to get all tangled in words.
I was looking for a good coding soundtrack, and turned to Spotify’s search for some quick wins. There were a ton of great lists, but Hacker’s Coffee was the only one that not only held me, but sent me looking looking for music from the featured artists. Highlights include Hans Zimmer’s Mombassa, from the Inception Soundtrack, a number of tracks by 65 days of static, and Infiltration, from the Mass Effect Soundtrack. Due to this soundtrack’s sheer depth (532 tracks), there is a high chance you’re going to find something you like.
A couple weeks ago, I randomly saw Steve Martin with the Steep Canyon Rangers. While I’ve never been a fan of bluegrass, I was super impressed by what I heard and felt. Turns out the twangy busyness of the banjo (which should be quite distracting), actually turns into a lovely mélange when backed by guitars and mandolin. Look out for all the awesome fiddle hooks, too, especially on the title track Radio.
]]>It’s been amazing to see how the opinion of open source has changed. Early on, it was a point of pride to say that one didn’t use third party code. It was viewed that if you did that, you didn’t know how to program.
As time went on, the industry’s perspective has shifted to the point where writing significant amounts of your own code is folly. The logic being that there are too many details that need to be accounted for to build a meaningful app. Any amount of time spent on problems that have been solved over and over again was time not spent on your product.
Nowadays, thanks mainly to Github, any problem that I’ve encountered can be (mostly) solved by a search through open Github repos. Many of these freely given solutions have been things I’ve used to build to a business. In most cases, I’ve used these solutions for free, without even thanking the author.
Now that my life is a bit different, it’s time to give back, lest karma catch up with me. My goal is to contribute to a project every week for the next year. While I have a couple good ones in mind, I’m totally open to suggestions. And to stay accountable, here’s a page I set up to track my contributions.
]]>It was amazing seeing students taking an interest in technology and coding. Many of them had already completed the Hour of Code projects. Others were still tinkering, seeing what they could add and change. I spent some time critiquing and reviewing work they had done in Mrs. Ramirez’s Web Design class. The students had all created functional portfolio sites, complete with content, small bits of javascript, and even Flash games.
The students were interested in my day to day, and the path I took to get where I am in my career. The students were amazed to hear that I didn’t work in a dark closet by myself, but in a brightly lit office constantly interacting with colleagues. They were also amazed to hear that the skills they needed to be truly successful in tech weren’t just coding and hardware, but communication, teamwork, and judgment. How a college education fit in to a career in Web Operations came up a few times. It was particularly interesting because I’m not aware of any four year university that offers a program in the field.
Students were particularly interested to hear about about some of the work I’ve done around security and incident response. Tales of defending against DDoS and spam attacks were a great way to let them know that career paths in tech aren’t just limited to coding or back office IT. They were spellbound to hear that mitigating DDoS attacks were actually something that could be part of an actual career path, and not just something they see in TV and movies. I truly hope that some students are inspired to pursue a career in Systems Administration and Web Operations.
One thing that was made clear to me as I was asked to describe my path to Web Operations was there is no clear path, and that sad fact has resulted in a shortage of people in the field. I would encourage other folks in Web Operations to spend some time talking to young students about the work that we do in the hopes that in a few years the industry will grow and mature. After all, for every class that learns to code, there need to be a few students who know how to run the infrastructure for that code.
]]>However, in a world where there could be hundreds of machines, this quickly becomes unscalable. Even with only a handful of machines, a team could spend a ton of energy and time simply remembering where the source of truth lives and how to decide which machine should be repaired next.
A team could write some code to pick a node and run the necessary commands. This is a bit better, but the requirement is that the command runs regularly on each node in the cluster. Without a central record to track which nodes have been repaired, there’s no way to know when and where the last repair was run, and no way to accurately predict which node will be picked next.
Enter AWS EC2 Tags. While EC2 tagging has some restrictions, it’s an ideal place to register small bits of metadata and completely replace our old school document. For our Cassandra example, it’s easy enough to use the EC2 APIs to list all machines in an Autoscale Group, or by tag, and then apply some logic to pick a machine.
Here’s some sample code that will look at the instances in an ASG, and pick one that either hasn’t been repaired, or the node repaired the longest time ago.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
The above snippet is based on the venerable boto library. Using python for interfacing with AWS allows you to use a mature, well-tested library. You can even write tests against code that uses boto with a library of mocks called moto. In short, it looks for instances that haven’t been operated on first. This is to cover the situation where you have a brand new cluster, or when the ASG group replaces nodes. Secondly, it sorts the instances by the date of the last repair ascending, which gives you the oldest node at the zeroth element of the list.
Taking this approach to managing infrastructure allows you create simple,
testable code with very few dependencies. Had last-repair
data been stored in
a local database, it would require the team to create and maintain that
database, as well as write and test additional code. The team is now free to
spend that time on other things. Without needing to worry about the integrity of
a persistent datastore, it also means that this script can reliably be run on a
very ephemeral instance. In fact, it can be run from anywhere. Scheduled Jenkins
jobs would be a good way to run this, but with the advent of
Scheduled Lamdbda tasks,
an Ops team can cut down even further on dependencies required to perform
maintenance tasks.
Was I entitled to bail early? Probably, but that wasn’t the right way to say it. I needed to leave. Despite handling the incident deftly, I still felt fucking awful, and in a dark, looming, nonspecific way. I needed to run from the firehose of problems, lack of solutions, apathy and anxiety. The incident in question was a days-long spam attack. It wasn’t my first rodeo with hackers, but this mitigation required bringing several groups together, which is always a challenge at a large organization. To add to that, there was a pronounced lack of resources from my own team. This attack came at a time when we were gearing up to launch a new product, and the team was completely focused on it. Even I had been intensely involved with ensuring the stability of a new, shaky platform. Being pulled away just added to my frustration.
I had already spoken with my manager that afternoon, told him I was tired. He kindly ordered me to go home, and take a vacation. I took the next two days off, calling in sick. This was the second time in a month I’d unexpectedly needed time off. At home, I crashed, played video games, watched a bunch of shitty movies my girlfriend normally wouldn’t palate, and generally just tried to avoid the real world. Physically, I wasn’t all that tired, considering I had only slept 9 hours or so over the past 3 days. But my mind and nerves were totally shot, and worse, I felt like I didn’t care, which I hated myself for.
In short, I burned out. Hard.
Burnout is a state of emotional, mental, and physical exhaustion caused by
excessive and prolonged stress. It manifests itself in ways including anxiety,
loss of motivation and confidence, and even degradation of physical health.
Burnout can be prevalent in organizations that promote hero culture and where
employees maintain a strong a sense of duty...
I lead operations at a small part of a large company, and I’ve been there a very long time. My mission is to ensure the company’s online presence is up and safe. Admittedly, my sense of duty to the mission has always been somewhat overzealous. In fact, much of my identity is tied up in the job I do. Those sentiments have truly been assets in navigating a treacherous path, but it seemed clear that they have become a double edged sword. Once I got bored with Xbox, I went through a journal I’ve been keeping for a few years.
In it, there were multiple notes and rants pointing to the simple fact that I have an unhealthy work-life balance. I found a long running list of songs to learn on guitar, but I haven’t picked up my guitar in several months. I have notes on code projects I started but didn’t make any progress on for years. There are links to museum exhibits I wanted to visit that closed long ago. Reading through, I also realized I don’t have any more than a handful of friends outside of work. All in all, it was a pattern of regret.
But also a lot of reasons to make some changes. Burnout is caused by a combination of internal and external factors, some of which can be managed. My biggest realization, looking through my past, is that I don’t need a vacation, I need a change in how I manage my energy with regard to my sense of duty to my work. I need to be more cognizant of how I feel when external factors rear their head, recognize signs of nearing the burnout zone, and back away. Presented without comment, are my signs of burnout;
Things I need to do to prevent getting to this place in the future;
The other hard reality is that for me to affect this change, I will need to attempt to manage the external factors. There are going to be some failures, but I’ll need to be OK with that, or I’ll have to make bigger changes. I truly love what I do, but now is the time for me to take some of myself back.
Wish me luck.
A few great burnout resources that helped me write this;
]]>As our team was searching for ways to accomplish this, we started using the ubiquitous and humble webhook to push information between systems. At first, we simply started pushing Pull Request comments into our Slack channel. This immediately increased our velocity, as we found ourselves seamlessly moving discussions from traditionally asynchronous mediums (Github Pull Requests) to more real time mediums (Slack, In Person).
Eventually, we started adding more sources to chat via Slack integrations. Our project tracking tool was a clear next step, as most of the information we were pouring into it was helpful to others. Team members reacting to status updates with pointers and questions was yet another boost to velocity of information. Including brief comments of intent and even full blown plans gave us a hook for correcting mistakes before they even happened.
Dense Communication - Extremely high signal to noise ratio textual
information that's automatically compiled from different sources and intended to
cross mediums.
The flip side of this was that looking through our chat was like drinking from a firehose. There was an immense amount of fragmented, context-less information flowing at any given time, so much so that it was easy to drown. The uninitiated considered our channel noisy, which is a fair assessment. Those who didn’t have access to the cornucopia of links ubiquitous to our channel had an even harder time trying to make sense of things. Some even believed the noise was distracting.
However, for those of us deep in the shit, we found that we were on the same page more frequently. In person communication became high trust, where one speaker could say the wrong thing and the listener could hear the right thing, thanks largely to shared context. We found ourselves responding to status updates made via integrations with helpful suggestions, requests to clarify, corrections, etc. Proposed plans were reviewed earlier, and noted blocks were busted more quickly. In general, we reacted more, contributed more, and as a result, had better outcomes.
To critics of these practices, we say that team productivity is more important than individual productivity. As humans walk down the streets of New York, there are a million stimuli, some gross, and some deadly. To say that people can’t read and filter a few lines of text is a cop out.
]]>But for some reason, AWS users seem to shy away from using autoscale groups for anything but satisfying elastic demand. However, in my experience, autoscale groups are a critical component of any highly available deployment. Even deployments of a single instance.
The utility of a service that does nothing but ensure that the desired number of
instances available is the basis for creating self healing infrastructure.
However, the name Auto Scaling
has created a widely held opinion that the
number of instances must regularly change to make use of the service.
This is simply not true.
For example, you may find a situation where you need to ensure high availability for an application only built to run on a single server. Or you may have a snowflake server, or an application where it’s not worth having more than one instance. Instead of creating a standalone instance, you can create an autoscale group with min, max and desired set to 1. As long as proper automation is in place, and the application is a proper 12 factor app, the autoscale group will ensure there’s always a single instance available.
As a design principle, any infrastructure created should have the means to heal
itself. Don’t be put off by the name Auto Scaling
, as the utility and value of
keeping a single instance running (that doesn’t need to scale) using a well
understood and easily accessible service is worth it’s weight in gold.
Therefore, it made a lot of sense to take some time to experiment with both Go and Rust. It was also a ton of fun to play with some new ideas. To get a grasp of the basics, I looked to the tried and true FizzBuzz. FizzBuzz is a great exercise because it forces you to loop, use control logic (if/else or case), and understand some basic math operators.
Rust bills itself as a language for systems programming. While I didn’t get far enough to determine if that was true, I did get far enough to determine that Rust is relatively difficult to work with, in that it requires a compilation step & a run step before you can see the outcome of your code.
1 2 |
|
It was also a bit awkward in that creating an effective code → execute workflow required installing Cargo to manage the very small package I created. Compilation was a bit on the slow side as well, considering I only wrote 25 lines of code.
1 2 3 4 5 6 7 8 9 10 11 |
|
Golang is an open source, statically-typed language that’s been designed to make it straight forward to build reliable and efficient software. That’s what it says on the tin, and happily, that was my experience. Go is simple to compile and run.
1 2 3 4 5 6 7 8 9 |
|
It was also fast. Almost 3x faster than rust. It wound up being slightly more
code, as Go requires you to name the package, and to import the fmt
package
just to print to stdout
. However, the experience writing it was quite nice.
Very clear errors appeared at compile time, which was helpful as a n00b.
Here’s my FizzBuzz implementation.
Overall, it was a good learning experience to dig into the next generation of programming languages. Rust seems promising, should I need to actually do system programming. But Go seems like a great candidate for building fast, portable tools.
]]>The people who push the buttons that cause problems should be fired. Heads should roll.
I was a bit taken aback. In my short career, I’ve pushed a lot of buttons
that have caused a lot of problems. What was said was certainly not a
personal condemnation, but a statement made in frustration. Frustration born out
of seeing human error
turn up as the root cause of most incidents, and not
enough improvement.
The first is that the person pushing the button is probably just following a run book. They might be brand new. The run book may have been given to them and they were told not to deviate. That’s the point of run books, after all, is to allow ops teams to operate safely with proven procedures.
The second is that mistakes and accidents happen, no matter how much preparation, automation and resiliency engineering goes into a service. It is inevitable that services will fail, and getting frustrated or angry about it is irrational.
For both of these problems, it’s extremely damaging to allow individuals to be blamed for an incident. It destroys morale to see a team member singled out. A rough lesson learned by one team member doesn’t always translate to institutional knowledge that helps prevent the next issues.
What we need to look for are things that can written down, codified and repeated to ensure the same problem is prevented. If a run book procedure turns out to be incorrect in a given set of circumstances, then it needs to be reexamined, by the entire team. If an automation does the wrong thing, it needs to be fixed. If a system allows actions to be taken that damage the effectiveness of the service, you need to ask why those actions are allowed.
The point is, there is almost always an aspect of the system that can be changed to ensure problems are avoided. Focusing on that will be way more productive than focusing on individual actions.
]]>Combining ruby’s builtin libraries to parse and manipulate configuration files with serverspec is a quick and simple win. We’re no longer bound by having to use overly complex, brittle regexes to ensure files are created correctly.
Here’s a few ways to pull in rubygems when writing serverspec;
include
Here’s an example of installing the inifile
gem in your spec_helper.rb
.
1 2 3 4 5 6 |
|
Here’s a few examples of using the ini gem we installed to make our tests better.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
Parsing config files with similar libs that would be consuming them in production provides a lightweight, implied method of testing that those files are valid. It’s also a straightforward, programmatic approach to getting values out of configuration files. And not just scalar values, but lists and arrays.
Here’s a more in-depth example, using rspec.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
Antemasque is a newish project from Omar Rodriguez-Lopez, from Mars Volta and At the Drive-In fame. While Mars Volta have long been a favorite for coding, thanks to the free form jazz and overall chaotic energy, Antemasque is significantly more straight ahead progressive rock. Busy guitars, with only a light coat of fuzz, push forward and provide a sonic landscape you can lose yourself in. Cedric Bixler-Zavala’s vocals are elegantly mixed in, riding right alongside the guitars. Most of album has a quick, steady drive, with only a quick slowdown for “Drown All Your Witches” and “Providence”.
I found this album through Crave Online’s Best Albums of 2014. I’m a huge sucker for a band with a big sound and a strong female front, having spent a lot of time listening to the Joy Formidable after seeing them at SXSW years ago. Ume (pronounced ooh-may) is usually a bit more of what I’m looking for, tempowise, in coding music. Most songs are a laid back andante, moved along by thrumming bass. What I like most is Lauren Larson’s silky, gently echoing voice over shiny, just shy of screeching guitar licks.
As creepy as this band’s name is, they are anything but. Their more recent album, All Hail Bright Futures, was contagiously happy and comforting, but that’s for another post. Gangs, on the other hand, is certainly non-threatening, but has a big enough punch to kick your ass even through Apple earbuds. Both albums were instrumental, and as such have been a big part of my Spotify rotation. The standout track for me is “Search:Party:Animal”, with a desperate, machine guitar riff driving away, punctuated by mountainous bass guitar hits. Interwoven throughout are quiet, but perturbed bass interludes that eventually culminate in the band coming together for builds that drop into high jitter. The whole thing ends in a midrange modulation turned into a samba-esque power chord riff with an abrupt but tasty ending. The rest of the album presents the same mountainous walls of guitar distortion.
The RX Bandits were a favorite of mine a few years ago, and Gemini, Her Majesty is their first record in five years. Again, big thanks to Crave Online for uncovering this one. What I like here is the relaxed, smooth, yet exuberant crooning over bright SoCal guitar riffs. There’s lots of forward movement, as most of the album has a quick pace, driving at times, but powerful riffs where the whole band hits together.
]]>You do things yourself when you need fine grained control of a resource, or require a deep understanding of a mission critical function. Can anyone do it better than you can? Even if they can, do you have enough control in the direction to get what you need? Does the time it takes to learn and execute in a given domain warrant the value it provides? Or could that time be spent on something else that would move the needle?
Dead giveaways are things the user sees. Design and product development can’t be outsourced, nor can the actual execution of your product. Those stay in house, no matter what. Your ability to find your voice and interact with your customers is the thing that makes your product unique. If you cannot execute on your own product, it’s a strong indication you’re not doing the right thing.
However, depending on your needs, there gets to be a long list of things that aren’t your product, but are still critical to running your business. Campaign and transactional email are great examples. Executing on mass email is extremely difficult because of defenses put in place by major ISPs. You need an IP with a good reputation, reverse DNS, a host that will allow massive amount of email to leave it’s datacenter, and the list goes on and on.
There are also implicit supporting requirements of running a service that you need to have but shouldn’t ever try to build. Project tracking, monitoring, graphing, and alerting are all examples of components that have executed extremely well by others, and can simply be turned on or deployed with minimal effort on the part of your team.
In the end, what’s critical is spending your energy on the things that actually create impact. Knowing the intersection of tradeoffs and priorities is what will keep your team on the path that gets the job done.
]]>A ground delay program is the operations equivalent of shedding load. ATC essentially stops allowing inbound flights to take off to allow for the extra time and care it takes for en-route flights to land safely. This is very similar to Netflix’s implementation of the CircuitBreaker pattern, as it allows the resource having trouble to recover while keeping flights en route at a minimum.
While I was stuck, I found there’s a couple really neat tools that can tell you a little about your chances of getting there on time.
The FAA is kind enough to supply information on delays here. However, I found the information to be presented here a bit disingenuous. The map continued to show my destination airport as green, although flights were actually being held at departure points.
They have a delay index where every destination airport is given a ranking of 0 through 5, 0 being totally on time, and 5 meaning go back to the airport bar and grab another beer. They also very helpfully track whether that index is trending up or down. These relate directly to how long a Ground Delay Program has been running. The longer that ground delay program runs, they more likely that index is going to going to trend up. They also provide a listing of flights to your destination airport that you can use to benchmark how delayed other flights and airlines are.
]]>We decided that we were going to spend a few days in Napa, so naturally, a B&B was a great choice. We wanted to have breakfast provided, a bunch of other folks to chat with about life, and a warm, comfy place to call home for a couple days. We found the Inn on Randolph via the Googlez, and were impressed by the comfortable looking rooms, so we booked. They offered a wine tour through Platypus Tours so we booked it. It seemed like a good idea for 2 folks who haven’t been behind the wheel in years to not be behind the wheel and drinking. That’s about all the planning we did.
Upon arriving, we were greeted warmly, given freshly baked cookies and the lay of the land. Both Karen and Stacey were immensely knowledgeable about navigating Napa, and were able to recommend great places based on how we felt. What’s more, it seems the Inn on Randolph has taken advantage of a great network of wine makers, restaurateurs, and tasting rooms to provide a great experience. Each morning at breakfast, Karen or Stacey would ask if we had plans, and if not, could they help. On their recommendation, they booked us into amazing experiences. For wineries they couldn’t book us into, they provided tasting cards. Much of the value of staying at the Inn is the advice and access (read: free tastings) they provide. However, expect that to evaporate into wine purchases, as the recommendations will quickly turn into opportunities to buy very unique and delicious wines.
The Inn itself is gorgeous. It has a warm, comfortable palette of dark wood, grays, and cremes decorated with Victorian furniture. The Inn has also paid close attention to creature comforts that make for a truly relaxing stay away from home. The bathroom floors were heated, which makes the Inn the most luxuriant place I have ever stayed at. The beds were the kind that hug you and don’t let go, with heavy comforters that make it difficult to leave. (The only way I was able to get up was knowing my feet wouldn’t freeze on cold floors.)
All in all, this was an incredibly warm, comfortable way to spend a few days in wine country. The Inn aims to send folks to places that will educate and treat them well, and to provide a delightful place to roost at once they’re done.
]]>Eggs Benedict is arguably the more complex of the two, given how hard it can be to poach a goddamn egg properly. Also, since it’s a breakfast, a bad eggs benny can put a serious damper on your day. However, despite the whole poaching challenge, I’ve rarely seen it mangled. There’s even room for quite a bit of variance. A bit of apple cider vinegar in the water can impart a tangy flavor. A few seconds can make the difference between a completely liquid yolk and a more viscous one. Then we get to the bread (soda bread being a unique standout, at Wilfie and Nell. This is the foundation of the dish, so it can really make or break it. For example, an overdone, rubbery English muffin can be so challenging to even the sharpest steak knife that you wind up shredding the whole meal. Breakfast should never be a workout. Hollandaise sauce is yet another canvas which can be painted on in endless ways. It accepts most seasonings surprisingly well. Dill is my favorite so far. Then you have the pig portion of the meal. Ham steak, streaky bacon, it’s all fair game.
The you have the burger, the old American stalwart. Again, super hard to screw up, but even harder to stand out. You also don’t have to wait in line at Umami Burger to get a good one. The blend of meat that goes into the patty (LaFrieda is king here), the cheese, bun all have a universe of possibilities. In my opinion, the more fat you start with in your meat, the better. Any burger that has short rib within 10 feet of it’s name is almost guaranteed to have a great flavor and texture. As pricy as it was, the $25 Black Label burger at Minetta Tavern was really something special. The patty there was made of prime dry-aged neef cuts. The choice of caramelized onions was awesome, as was skipping the cheese, as the patty stood completely on its own. Cave aged cheddar, which Peels employs, has made for a notable meal. And of course, the bun is there to keep your fingers (relatively) clean, or just fall apart. It doesn’t even have to be a traditional bun. Whitman’s makes a patty melt that comes between two slices of a Blue Ribbon Pullman loaf.
The best thing about these two dishes is that they’ll never get old. As long as restaurants keep experiments, coming and going, there’s always going to be an awesome variation!
]]>After seeing many of these paths, and knowing that each one looks different to everyone, the only way to truly determine the wrong path is to wholeheartedly walk down a path as if it were the right one. My favorite learning experiences have been when I pursued paths that seemed right, but were not. When determining the right path to take, sometimes the best thing to do is pick a path, walk down it, and see if you get where you need to go.
Here’s to walking down each path as if it were right.
]]>The alerts will trickle in at first. It’ll just be a web sever or two that’s squawking. Then more. Then external monitoring will go off. Pingdom will mark you as down, a painful insult to your hard work, and numerous nines. Then all of the web servers will alert as down. And those alerts will keep coming. For a large infrastructure, potentially hundreds. You’ll have to quit email, or turn off notifications, or the cacophony of dings and vibrations will rattle around your brain and wrestle away whatever modicum of clarity you may have. SSH hangs, pings fail, your jump server gets squirrelly, and panic mounts. Tell your boss to get on chat. Don’t email, text, or call, because those channels will be fucked, occupied by automated alerts, hosting providers, vendors, and other team members.
Vendors might be confused because they can’t get to key pieces of infrastructure. That infrastructure and networking gear might be shared with others, which will freak everyone out more. If the attack is large enough, they may be experiencing the same feelings. You feel shitty, knowing this affected others, but you feel it later, because you can’t possibly feel more feelings. Hopefully your vendors have dealt with this before. Hopefully they know what they’re doing. Hopefully they’ll have an update in a few minutes. Hopefully that update won’t be that they null-routed your IP.
When the attack is several times larger than your subscribed bandwidth, service is denied. To your customers, to your team, and probably to upstream infrastructure. ISPs can’t, and sometimes wont, help. In fact, they’re likely to shun you. The dreaded null route, where there’s no hope of coming back up anytime soon. That’s the moment when there’s nothing you can do because the infrastructure leading to your environment has been overwhelmed.
A DDoS never happens at a convenient time. They happen late at night, during a team outing, or when you just took a sleeping pill, or have a major launch to contend with. The randomness is just another thing that makes you feel helpless, and ineffective. Impotent.
Once you’ve picked up the pieces, and mitigated, or waited it out, that’s when exhaustion comes. Or rage, then exhaustion. When and if you finally get to sleep the night after the attack, it’ll be fleeting. You’ll wake up often, check your phone, looking for alerts or a sign that it happened again. You’ll wake up early the next day, nowhere near rested, go back to the office and wait.
Incidents beg the question “who would do such a thing?”. Particularly when during your tenure, you’ve never done anything to hurt anyone, and your service was designed to help people. Hell, even your competitors even respect you. The whole thing was birthed to help people, the little people, the people who always get taken advantage of. You love this site on some level, or you wouldn’t be part of the response. If you don’t love it, you at least feel a duty to protect it. If you don’t fall into either category, and are blissfully unaware, or asleep, you’re a either a prick or incompetent.
Once it’s over, you constantly fear they’ll come back. Thoughts and theories swirl about, unchecked. People will ask about the perpetrators, and you can only shrug, wearily. In the vacuum of facts, folks will supply their own theories. The conjecture about the attackers will be endless, the convo du jour.
Conjecture is useless. The attacker doesn’t have a face. And even if it does, it’s extremely unlikely you’ll ever see it. The sad fact is that there will be no Liam Neeson Taken-style vengeance. You won’t suddenly appear on another continent, infiltrate their lair, and punch their teeth down their throat. No matter how bad you may want to.
When you get into the office again, you’ll likely be hailed as a hero. After all, you “fought the hackers.” Deep down, even after you got everything back in place, you still feel a little bit like a failure, because it happened at all. You take the congratulations sheepishly, through bleary eyes, deathly afraid of the next attack.
Pragmatically, most of your energy should be focused on things like re-ip'ing your web site. Engaging with hosting providers to get mitigation services in place, talking directly to mitigation experts to see what help they can be, analyzing the attack to see if there are ports that can be closed to an attacker completely, fixing your application to handle the closing of ports that were attacked on, standing up services to get around ports being closed, altering your application to gather inputs in a more intelligent way after the hacks that were put in place assuming there would never be any serious infrastructure changes made without seriously considering them first.
You have a lot of work to do.
]]>