Cargo Cult System Administration

by Bob Plankers on October 19, 2011 · 7 comments

in System Administration,Virtualization

The term “cargo cult” comes from World War II, where the USA and Japan created small airstrips and bases on islands in the Pacific Ocean, and the locals mistook the cause & effect in relation to why goods and cargo appeared on the island. It’s absolutely fascinating, and the Wikipedia entry for cargo cults indicates some of these cults are still operating.

The excellent book Code Complete, by Steve McConnell, introduced me to the idea of cargo cult software engineering. Turns out, a guy by the name of Eric Lippert coined the phrase “cargo cult programming,” and explains the whole thing well:

During the Second World War, the Americans set up airstrips on various tiny islands in the Pacific.  After the war was over and the Americans went home, the natives did a perfectly sensible thing — they dressed themselves up as ground traffic controllers and waved those sticks around.  They mistook cause and effect — they assumed that the guys waving the sticks were the ones making the planes full of supplies appear, and that if only they could get it right, they could pull the same trick.  From our perspective, we know that it’s the other way around — the guys with the sticks are there because the planes need them to land.  No planes, no guys.

The cargo cultists had the unimportant surface elements right, but did not see enough of the whole picture to succeed. They understood the form but not the content.  There are lots of cargo cult programmers –programmers who understand what the code does, but not how it does it.  Therefore, they cannot make meaningful changes to the program.  They tend to proceed by making random changes, testing, and changing again until they manage to come up with something that works.

I see this all the time in IT, so much so that I refer to it as cargo cult system administration[0]. Staff who insist that “it just needs a reboot” to fix whatever is wrong, perhaps a locked file or a PC that’s slow. Staff that dig around in the Windows Registry and use “cleaners” to make it all better. Staff that have no idea how networks work, but end up disabling firewalls or monkeying with settings like the MTU because they saw someone do that once and it fixed the problem. These people may fix the problem, eventually, but they do so while creating all sorts of collateral damage. God help them when they need to make big changes to systems.

How do these people keep jobs in IT? I expect this level of ignorance from the general masses of users, not from their support staff. But that’s the problem. The folks hiring these people don’t know, either. They’re part of the general masses of users.

Nowadays, with systems as complex as they are, virtualization and storage and networking and servers being all tied together, you need IT people that, at the very least, don’t think it’s all mystical. And perhaps the solution to finding a person like that is to ask a good IT person to help you hire. Kick them some cash to help you screen resumes and interview a couple of people. You’re looking for a specific kind of IT magician, after all, the kind that can pull excellent solutions out of hats. Not the guys that don’t understand why the planes stopped landing on your island.

 

[0] Update: The name may have come from Michael Janke’s post, “Cargo Cult System Administration,” back in 2009. Sorry man, apparently missed that when I searched for cargo cult stuff, and I know I read the post (and probably linked to it here).

—————————————

{ 7 comments }

Joern October 19, 2011 at 12:59 PM

Look at wikipedia for Dunning-Kruger effect this sums it up pretty much.

Mitch Wright October 19, 2011 at 2:46 PM

So true! For example I’m not sure whether to laugh or cry when interviewing an “experienced” Linux system admin about adding users to systems. All know how to use one UI or another, but many have no idea what files are modified or how they relate to one another.

Matt Simmons October 19, 2011 at 3:23 PM

It’s also worth checking out Michael Janke’s article on the subject:

http://blog.lastinfirstout.net/2009/11/cargo-cult-system-administration.html

Bob Plankers October 19, 2011 at 3:46 PM

Oh crap, I hate when I inadvertently rip people’s ideas off. I knew I’d seen it before, but I didn’t know where I got the term and figured it probably just came from me reading Code Complete or something back in the mid 2000s. Didn’t show up in a cargo cult Google search. I updated the post, good catch. Sorry Michael.

Alex S October 19, 2011 at 4:37 PM

I think it’s like Layer 9 of OSI model: the Religious Layer, one layer above “political” or “user”. That’s when you can’t explain anything and rely purely on dogmas and prayers.

Michael Janke October 19, 2011 at 6:34 PM

Great minds think alike. ;-)

Brian October 20, 2011 at 8:31 AM

People who operate this way are able to continue because they do actually deliver something. Whether it’s good enough to meet the standards of another senior technologist is irrelevant to the business user who is receiving the result. The them, ALL IT is just some sort of magic, and they cannot discern the difference.

Comments on this entry are closed.

Previous post:

Next post: