Cargo Cult System Administration

The term “cargo cult” comes from World War II, where the USA and Japan created small airstrips and bases on islands in the Pacific Ocean, and the locals mistook the cause & effect in relation to why goods and cargo appeared on the island. It’s absolutely fascinating, and the Wikipedia entry for cargo cults indicates some of these cults are still operating.

The excellent book Code Complete, by Steve McConnell, introduced me to the idea of cargo cult software engineering. Turns out, a guy by the name of Eric Lippert coined the phrase “cargo cult programming,” and explains the whole thing well:

During the Second World War, the Americans set up airstrips on various tiny islands in the Pacific.  After the war was over and the Americans went home, the natives did a perfectly sensible thing — they dressed themselves up as ground traffic controllers and waved those sticks around.  They mistook cause and effect — they assumed that the guys waving the sticks were the ones making the planes full of supplies appear, and that if only they could get it right, they could pull the same trick.  From our perspective, we know that it’s the other way around — the guys with the sticks are there because the planes need them to land.  No planes, no guys.

The cargo cultists had the unimportant surface elements right, but did not see enough of the whole picture to succeed. They understood the form but not the content.  There are lots of cargo cult programmers –programmers who understand what the code does, but not how it does it.  Therefore, they cannot make meaningful changes to the program.  They tend to proceed by making random changes, testing, and changing again until they manage to come up with something that works.

I see this all the time in IT, so much so that I refer to it as cargo cult system administration[0]. Staff who insist that “it just needs a reboot” to fix whatever is wrong, perhaps a locked file or a PC that’s slow. Staff that dig around in the Windows Registry and use “cleaners” to make it all better. Staff that have no idea how networks work, but end up disabling firewalls or monkeying with settings like the MTU because they saw someone do that once and it fixed the problem. These people may fix the problem, eventually, but they do so while creating all sorts of collateral damage. God help them when they need to make big changes to systems.

How do these people keep jobs in IT? I expect this level of ignorance from the general masses of users, not from their support staff. But that’s the problem. The folks hiring these people don’t know, either. They’re part of the general masses of users.

Nowadays, with systems as complex as they are, virtualization and storage and networking and servers being all tied together, you need IT people that, at the very least, don’t think it’s all mystical. And perhaps the solution to finding a person like that is to ask a good IT person to help you hire. Kick them some cash to help you screen resumes and interview a couple of people. You’re looking for a specific kind of IT magician, after all, the kind that can pull excellent solutions out of hats. Not the guys that don’t understand why the planes stopped landing on your island.

[0] Update: The name may have come from Michael Janke’s post, “Cargo Cult System Administration,” back in 2009. Sorry man, apparently missed that when I searched for cargo cult stuff, and I know I read the post (and probably linked to it here).

—————————————

• Eric Lippert’s original blog post introducing the term.
• Steve McConnell’s essay on Cargo Cult Software Engineering.
• Wikipedia’s entry on “cargo cults.”