The Lone Sysadmin

“We had two bags of grass, seventy-five pellets of mescaline, five sheets of high-powered blotter acid, a salt shaker half full of cocaine, a whole galaxy of multi-colored uppers, downers, screamers, laughers. Also a quart of tequila, a quart of rum, a case of beer, a pint of raw ether and two dozen amyls. Not that we needed all that for the trip, but once you get locked into a serious drug collection, the tendency is to push it as far as you can.”

“How long could we maintain? I wondered. How long until one of us starts raving and jabbering at this boy? What will he think then? This same lonely desert was the last known home of the Manson family; will he make that grim connection when my attorney starts screaming about bats and huge manta rays coming down on the car? If so, well, we’ll just have to cut his head off and bury him somewhere, ’cause it goes without saying that we can’t turn him loose. He’d report us at once to some kind of outback Nazi law enforcement agency and they’ll run us down like dogs. Jesus, did I say that? Or just think it? Was I talking? Did they hear me?”

Once upon a time there was a man who was a sysadmin. He took care of twenty IBM AIX boxes. These machines were built by his predecessors, and they were built over time, each different, each with a personality. This man’s customers didn’t like the different personalities. They wanted each server to be the same so they could write scripts that had the same results on all the servers. They wanted to compile and not have to worry about differences between compilers on the machines.

This man asked around and discovered that IBM’s Network Installation Manager, or NIM, was a tool he could use to manage the machines. He took an older AIX box and made it a NIM master. He started using the tool to patch and upgrade older machines and install new machines. The process he used to manage his twenty servers was involved and inflexible. On several occassions he inadvertently destroyed machines using NIM, instead of upgrading them. New procedures had to be used to take image backups of the machines prior to his work in case something went wrong.

Many of his days were spent maintaining the tool, rather than maintaining the servers. The days he didn’t maintain the tool itself were spent on corollary tools to fix problems caused or exacerbated by the primary tool. The time spent on these tools far outstripped the time needed to admin the twenty servers by hand.

Once upon a time a manager wanted a better way for his group to keep track of work requests. Things were getting lost or delayed sometimes because his staff was getting distracted with new walk-up requests. He was familiar with the enterprise case tracking system his organization’s help desk used (Clarify ClearSupport), and decided that all work his group did should be entered into that system and assigned to an individual.

After a few weeks he examined the records and discovered that according to the case tracking system nobody in his group was doing any work. The few cases that were entered by people were woefully inaccurate and included very few details of the work done. He demanded that everybody enter everything they were working on, and do so accurately.

A week later he was checking the time reports from his staff and discovered that nearly all of them had spent many hours on “miscellaneous tasks.” When he asked his staff why they were spending so many hours on miscellany they replied that it was the recording of their work in the case tracking system that was taking so long. The act of doing the paperwork for their work was taking longer than the work itself.

Both of these cases suffer from the same problem: powerful tools. IBM’s NIM is a very powerful tool for managing machines. ClearSupport is a very powerful case tracking system. These tools are used very successfully by many large companies to manage their enterprises. In both cases, though, they are too big for most endeavors. NIM is great for hundreds of machines, but it is overkill for twenty. ClearSupport is wonderful for recording and escalating cases to different parts of a big organization, but is overkill as a shared to-do list.

When I am evaluating tools I have a series of questions I ask myself about that tool. I think of tools in terms of return on investment. If my team invests 10 hours of time in this tool will it save us more than 10 hours of time? Will it do so rapidly?

Does the tool help achieve other goals? A break-even ROI might be worth it if you can accomplish multiple goals or significantly improve the quality of your work. Task management systems often can help with customer billing and time reporting. Patch management tools might help with audits and compliance checking. Like using a pocketknife as a screwdriver, sometimes one tool should not be forced to take the place of one more suited to the task.

How much will the tool cost? Could I just hire someone to do the same job, or do it in a different way with a low-tech solution? ClearSupport is tens of thousands of dollars. Tasks Pro is $500. A white board is $100. Could I write the tool myself?

How much time does it take to maintain the tool? How does that compare with just doing the work manually? An example of this would be using NIM for patching. How much work goes into using NIM versus downloading the AIX patches and using the native package management tools to update the machine?

Does the tool require me to change my behaviour, or can I continue using my previous habits with this tool? Teams should be flexible enough to change their habits, but you should also choose a tool that can work with your setup. NIM is great when you have a consistent maintenance window and lots of machines that can be updated in lockstep. It isn’t as shiny when your servers all have different maintenance windows, and are all a little different here and there. If your situation doesn’t match the design of the tool then move on.

What will I need to learn to operate this tool? Does it use its own language or configuration syntax? Will I need another server to run it? What happens when the tool breaks? If I cannot fix it right away is there a workaround? Can I start using this tool a little bit here and there and work into it, or do I need to dive completely in to make it work? Did I really need the feature that is missing from this tool? Does this tool’s integration with other systems matter? Will we actually integrate it, or is that just an excuse? Would it be better to have two tools that each do their job very well, or one that is merely okay at both?

Sometimes these are tough questions. Sometimes you have to fight your ego and the urge to pick a big tool when you really just need a small one. Sometimes you have to fight your management and your coworkers when they cause scope creep, evaluating tools with criteria that make no sense. “The patching tool we choose should have account management features.” Um, what? Hell no. Sometimes you just need to search freshmeat.net and call it a day.

Your tools should amplify your productivity, not rob it. Whatever you choose think ROI.

• AOL Stalker - The leading resource in anti-privacy
  OMFG
• Something Awful: The Internet Makes You Stupid
  If I were an AOL user right now I’d be searching for “class action lawsuit.”

Oh, hell yeah. I hope they do the original Teenage Mutant Ninja Turtles justice. The trailer looks amazing, insofar as the graphics look wonderful.

The end of the trailer has “Only in Theaters.” Have they heard of the Internet?

For those of you reading that may have listened to my rant a couple weeks ago about the guy in the bar that wouldn’t shut up about the Transformers… um, this is different. That guy was speculating what kind of people Optimus Prime and Starscream would be if they kicked up their heels and smoked a big fatty. Like, WTF, I’m cool with the abstract thinking but not when you never shut up about it.

And that’s all I’m gonna say about that.

Just saw “A Scanner Darkly.” I’ve seen bad reviews. I’ve seen good reviews. It is good. Linklater didn’t make it funny as some have criticized, though parts of it are. Humor happens sometimes even amidst bad situations.

“What does a scanner see?” he asked himself. I mean, really see?

Because, he thought, if the scanner sees only darkly, the way I myself do, then we are cursed, cursed again and like we have been continually, and we’ll wind up dead this way, knowing very little, and getting that little fragment wrong too.

I can’t remember OpenSSL options. Having grown tired of looking up how to create a self-signed certificate and finding lengthy tutorials I have opted to write my own. Three easy steps to having your own completely kinda-trustworthy certificate for testing and whatnot:

openssl genrsa -out bogus.key 1024

openssl req -new -key bogus.key -out bogus.csr

openssl x509 -req -days 365 -in bogus.csr \
-signkey bogus.key -out bogus.crt

There, that was easy, wasn’t it?

Update: NickyP commented with a one-liner for this. That’s why I love blogs. Thank you!

“Bob, why do you always complain when people encrypt their web server’s SSL keys with DES3?”

Well, since you ask, it’s because the web server needs human interaction to start. You need to supply the password.

To remember the password you need to write it down somewhere. Or store it online so you can retrieve it. These are potentially insecure, they are avenues of compromise, and at the very least they create more work for already busy administration teams.

You could set the password to be the same on all the servers, and tell everybody what it is. That might solve the administration problem but negate any value from encryption.

Having a password means you cannot background the web server startup process. You can’t have the web server automatically start, so you can’t have a machine that just fixes itself on boot. One of my main tenets of large-scale system administration is that a system must make itself right on startup. If the machine boots but doesn’t immediately begin doing its job it is faulty. Any process that requires human interaction does not qualify as automatic.

I’m all for defense-in-depth, but not when it sacrifices so much to gain so little. Who would have access to the server key? The admins, and maybe the users on the machines. The users on my machines generally share the same positive goal as I, and I don’t give accounts to those I don’t deem trustworthy. I fight problems with hacked accounts with firewalling and password checking and policy and user education. I can fight more potential corruption by ensuring that the keys are properly chown’ed and chmod’ed 400.

What am I fighting with -des3, aside from myself? Someone give me a good example, please.

• Lieberman’s ‘hack’ was no such thing
  More proof that Lieberman should not be in office.