The Technician

Small, obscure optimizations sometimes have the potential to make the greatest impact. For example, every time a connection is made, MySQL will do a DNS lookup of the host that is trying to connect. If MySQL is handling many connections, the overhead of an extra DNS lookup can be hefty, simply because of the number of extra operations that have to be performed before MySQL can actually start doing actual work.

Thankfully, there is an option in recent versions (4.1+) of MySQL that will instruct MySQL to skip the extra DNS lookup. It’s a fairly obscure option called skip-name-resolve. The only caveat to using this option is that the users defined the GRANT tables can only use IP addresses as hostnames. For most MySQL users, this shouldn’t be an issue.

Setting up a sandbox environment has normally been a trivial task. Set up a vhost, get a copy of the database, build out the app, and start doing stuff. When ‘Stuff’ is ‘Done’ push the changes onto production, and bask in your own crapulence. That is, until your data set exceeds the limits of the sandbox, and the SOA which is saving the day in production is becoming nothing but a headache in development.

The basic sandbox environment for an app includes a reasonably recent data set, similar ( underpowered can be OK, depending ) hardware, the exact same versions of php, mysql and apache. Php, mysql and apache need to be configured exactly the same way as in production. In fact, as part of this process, it might be useful to pull down those ever so important configuration files, put them in a safe place. Perhaps source control (cough). Consistent configuration is extremely important. Bugs produced by configuration problems are notoriously hard to reproduce, and result in devs combing through code looking for bugs that don’t exist.

Maintaining a few sandboxes should be a trivial endeavor. That is, until your project gets too big. A natural response to handling ever-growing problems is use a Service Oriented Architecture; that is, to shard off aspects of the app and dedicate hardware and resources to it. However, three or four shards later, multiplied by an environment for each developer, the guy who was doing sys admin work as needed just became full timer. Unfortunately, there’s no way around this, even with a clever sys admin, who can leave enough automated scripts around so that developers can *mostly* maintain their own environment.

The fact of the matter is maintaining the development environment is one of the most important things a company can do. Close attention to detail in the sandbox will make all the difference in the deployment process. Changes in code base, file permissions and configuration can all be tested and deployed the same as in production. So every build to every sandbox (everyone builds daily, right?) is a chance for the development team to catch mistakes, and learn from them, before the big push. And if that fails, we all know how to handle a crisis.

Everybody procrastinates. Some tasks that get pushed off don’t matter, it just gets done later. Some tasks that go over deadline result in profanities and bloody noses. But every once in a while, tasks come along that have an expiration date. As in, if it doesn’t get done by a certain time, it doesn’t matter.

You ever heard of a “Japanese Inspection?” Japanese Inspection, you see, when the Japs take in a load of lettuce they’re not sure they wanna let in the country, why they’ll just let it sit there on the dock ’til they get good and ready to look at, But then of course, it’s all gone rotten… ain’t nothing left to inspect. You see, lettuce is a perishable item…like you two monkeys.

– Big John, Days of Thunder

What Big John was referring to was the fact that all he had to do was ignore Cole and Rowdy until they didn’t any fight left in them. Tasks can be just the same way. Eventually, the need for the task to be done just goes away, or starts to smell. The only thing that really matters is being able to tell the difference between the things that really need to get done, and the things that just aren’t important enough to get done.

For a very long time, I was convinced that a FULLTEXT index in MySQL was the best solution for all your searching needs. Then I realized that it was horribly slow, and mixing with complex joins completely destroyed any chances of using MySQL indexes in any way that would make sense or get decent results. The solution to fast and scalable free text search on any website is, of course, a Full Text search engine.

There are a few different ones out there. After a brief affair with Lucene, I settled on Sphinx. Sphinx is easy to install, even on 64-bit machines, and is architected in a way that makes a lot of sense for the web. The following steps were performed on a Red Hat machine. Don’t skip the mysql-dev install, even if you already MySQL installed.

> yum install gcc-c++
> yum install mysql-dev*
> wget http://www.sphinxsearch.com/downloads/sphinx-0.9.9.tar.gz
> tar xzvf sphinx-0.9.9.tar.gz
> mkdir /usr/local/sphinx
> ./configure –prefix /usr/local/sphinx –with-mysql
> ./make
> make install

Once installed, it’s fairly simple to start playing with the packaged example data and queries. The php APIs make integration easy, either to build a service, or use locally as a substitute for MySQL. In fact, as long as the index can be kept reasonably up to date, Sphinx is a better choice for complicated sorts than MySQL.

Having your own domain and server is a really fun thing. You can keep a blog, store files, post photos. All good ole down-home interwebz fun you can dream up. With the advent of micro-blogging, everyone now has an audience they need to communicate with in the briefest fashion.

Short links have become the norm for sharing, but with a huge price. Short links break the interconnectedness of the Internet. Search results that depend on the count of links become incorrect. Some services try to reconnect links between domains by using some DNS trickery. But the issue remains that there is a middleman that can’t help make the connection.

Unless, of course, that middleman is the search engine. Google’s recently announced URL shortener could solve many of the problems inherent with URL shorteners. By being the middleman, Google would have all the necessary information to put the pieces back together. It’s easy enough to set up, provided you have a Google Apps account. It does not currently have an API, but here’s hoping.

30 Seconds to Mars recently released their third album, This is War, and it is quite the departure from their first album. Brand New Day was raw and angry, with amazing guitar sounds, great composition, and a real sense of urgency in the writing. The album was really exciting to listen to, and the live performances were great. I saw them at Avalon in NYC a few years ago, and it’s still one of my all time favorite shows.

This is War is mild and boring in comparison. The effects driven distorted guitars characteristic to Brand New Day is missing. They seem to have been replaced with over-produced electronics. Jared Leto has less than half the intensity than he did in Brand New Day, and the lyrics have lost their edge. The focus on deeply layered choruses of what sounds like children singing lacks impact. The collaboration with Kanye and his 808 didn’t really go anywhere, and just seemed to pull the band further from their roots. It’s sad to hear a band that had such a great band with a unique sound has gone so far off track.

Occasionally, a project will come across my plate with the criteria, ‘Make sure this works everywhere, is completely template-able, and is something we can grow with.’ Normally this is coupled with ‘We need this to work with X *right now*, and Y and Z later.’ What I really hear is ‘Make it work for X, and ship the damn thing.’ After all, hitting those deadlines is really important.

Of course, this has a whole bunch of ugly assumptions tied to it. The first is: when I get to Y, everything I did for X will work. All I need to do is drop in a few config changes, and tweak a few parameters, and I’m done. (Yea, right) Secondly: that every case Y needs to cover is contained within X. (Not Likely) Third: All of this will be so well documented that any literate individual will be able to implement Y by osmosis.

So. Do we spend time now or later? Shipping X seems pretty simple, so why not just build X, satisfy the business dude and call it a day. Spending time now means that deadlines may have to shift, and something that should be simple becomes complex. We have other, more important, projects to work on.

Eventually Y comes calling. So let me introduce you to…Future Web Dev Guy Person Girl! If you’re lucky, that person is you. If you’re not, it’s another dev. The assumptions we made back in paragraph 2 have reared their ugly heads. Since they were assumptions, you’re probably boned. If not, you’re probably one of these guys. If you’re the rest of us, Future Web Dev Guy Person Girl definitely hates your guts, because the groundwork that was supposed to be laid out is not there. They’re running through a lice-infested rat’s nest of procedural functions trying to pass the additional variable that will make this all work.

The best way to keep Future Web Dev Guy Person Girl from cursing like a sailor is implement correctly, test thoroughly, and deal with Y before it’s due. Deadlines need to be managed according to project scope, and if project scope includes Y, it needs to be accounted for now, before you lose a friend in Future Web Dev Guy Person Girl.

I am pleased to announce that this site is now hosted on the Rackspace Cloud. It was a simple migration from MediaTemple, and has given me the level of control I want. I got to choose my OS (CentOS), versions of php and MySQL, and setup apache how I like it. I’m free of Plesk and those and the limitations therein.

The one thing I would really like to see from the Rackspace Cloud is DNS Support. My goal when migrating http://chr.ishenry.com was to move entirely off of MediaTemple. The one thing I really did like about hosting with them was that DNS was integrated directly into the service. With the Rackspace Cloud, there was no such convenience. However, a quick signup with DynDNS and a tweak to my domain registrar solved that.

Big thanks to Ryan Kearney’s video tutorial for the yum command that brought everything together.

[Update] My team at Rackspace has sent me the fluffiest, most comfortable pillow I have ever had.

When a hosting provider goes down, there are lots of questions that get raised. Is my host reliable? Will they flake out during crucial times when my site needs the traffic? Will they double bill me?

Since I have been working with Rackspace, they have had less than stellar uptime, with issues mostly related to power. My company pays a lot for hosting with them, and downtime for a young company is deadly. But oddly enough, I’m still OK with Rackspace hosting my company’s myriad services. The benefits of hosting with them have been so great that a couple hours of downtime is nothing.

First off, their SLA has provisions for downtime, when it happens. If your server has a legitimate issue, you’re entitled to ask for a credit. To me, this is a promise that they’ll put their money where their mouth is. And if you call them on it, they’ll be reasonable.

Secondly, their support during crises is still amazing. During the truck incident, I was able to get a tech to run fsck on my disks, and hang out to watch no questions asked. No, I am not on their intensive plan.

Third, their support culture is simply amazing. Their linux techs are always willing to look deep into an issue to find a resolution, and they provide much of the basic infrastructure that is hard to come by for small companies.. They’re also completely willing to educate their customers about the servers they maintain.

In short, Rackspace has been the target of a lot of criticism over issues in their datacenters. The fact of the matter is that there will always be issues and downtime. Their SLA guarantees the impossible, which they seem to realize, as any failure on their part comes with swift response. In the end their SLA seems to be more of a way of setting standards than anything else.

[Full disclosure: I haven't slept in 2 days because of their power issues]

Sometimes bugs come along that require significant work to fix. Depending on what project timelines are like at the moment, sometimes fixing the bug isn’t the best option. For example, a race condition in the caching architecture causes pages to be stale. The persistent data store is correct, but the cache is not. To the person who just triggered the update, there’s a bug. The information on the public side is not in sync with the information they just entered.

So, like any other bug, a report will eventually percolate down to the dev team. People scream, fortunes are lost, the svn blame command is used, and the devs who wrote the code pee their pants. Once the chaos dies down, the actual prognosis of this issue can turn out to be extremely grim.

A shortcoming of the caching architecture shows that there’s a race condition when the system is under heavy load. In order to fix it, the dev team needs to plumb the depths of the data access layer, and probably change some parameters. But that’ll probably break everything. Everywhere. Or the layer manipulating the data could be fixed to replace the cache instead of invalidating. Except the methods to manipulate that entity live in 3 different codebases. It’ll probably break the editor. Either way, the actual solution doesn’t matter.The dev team certainly needs to do something, and it needs to be released three days ago.

The correct way to fix this issue will vary widely depending on circumstances. But in this particular case, the best answer was to not fix it, just manage it. Our team was busy, there were other projects that were more pressing. Plus the codebase was being rewritten. So instead of flogging a dead horse, a simple script was thrown together that compared the cache and the database. If they were out of sync, the cache would be cleared, and would be repopulated with the correct information the next time it was requested. Once it was implemented, the bug was still there, but the cache seemed to be up to date.

Every dev team will face bugs that have enormous costs to fix. The way to deal with these bugs will be different every time they come up. It’s important to remember that managing bugs can be almost as effective as fixing them.