The Lone Sysadmin

I’m exhausted. In the last five days I’ve spent 26 hours at the Bartell Theatre in Madison, along with two days of weddings, and softball practice (I haven’t played softball in years and my arms are going to fall off). I do a lot of work as a tech with the Strollers theatre company. I’ve looked for a hobby for a long time that involves a minimal amount of computers but still appeals to my technical nature, and theater tech work fits the bill. Often the tech work spills over into other non-Strollers stuff, like installing a new stage floor, but it’s fun, and the techs get comp tickets (usually used by friends, since the techs often see the show many times). I’d recommend it to anyone looking for something to do once in a while on a Sunday.

Anyhow, I’m off to the Twin Cities, then to New York City to see family, friends, and the Minnesota State Fair. Going to the fair is more of a gastronomic event than anything, as there’s so much junk food to eat. You could describe the course my brother and I always take as: malt, foot long hot dog, jerky, jerky, sno-cone, mini-donuts, cheese curds, mini-donuts, water, water, cookies, milk, milk, milk, milk, milk (the Dairy Council runs an all-you-can-drink milk stand), milk, milk, taco, taco, milk, milk, ugh — waddle home. And all while you’re looking at cows, horses, tractors, cars, news stations, etc. I love that place.

When I get back my first order of business will be to find a new layout for this site. I really like what A List Apart has done with their site, and it’s inspired some ideas for a new look here. Hopefully the flight and my NYC adventures will cause a good design idea to issue forth.

There seems to be a spectrum that all people can be classified into. One end of the spectrum is “doing” and the other is “planning.” There are the cowboys at one end, who never plan anything but just do things. There are the dullards at the other end that never actually do anything because their plan might not be complete.

Most of the IT folks I work with seem to fall in the center somewhere, where they know just the right amount of planning that it will take to implement something, but then can switch gears and get it done. Often these people fall towards the “doing” side of the spectrum. I’ve noticed a disturbing trend lately, though, where managers are pushing their people to be on the planning end of the spectrum. This is great when the people they’re pushing are cowboys, who just do things and make changes without care or regard for anybody else. But for a lot of people these extra planning steps just reduce their effectiveness, and ultimately end up causing frustration. They’ve already done the planning, and are caught in two poor assumptions their managers have:

1. Meetings, and meetings alone, constitute planning.
2. In order to have effective planning for a change you must ask everybody who has anything to do with anything you’re changing.

I work with a lot of highly skilled techies, who, when they want to change or do something, can work out the plan in their head by themselves. They walk around to all their coworkers and use them as sounding boards first, then approach the stakeholders and talk to them about it. Once people agree they go ahead and implement that plan. They fill out all the paperwork in their change management systems, typing in a well thought out, cohesive plan that took all of 20 minutes to think up and get support for (and then another 20 minutes to document). They schedule the change for a time that’s least disruptive to everybody but themselves, just in case something should go wrong.

Suddenly, though, these techs that have pulled off change after change after change, flawlessly, are being forced to hold meetings and build consensus with people who have no freaking clue what the tech is talking about. If a meeting was necessary they would have held one, but if it wasn’t really necessary they wouldn’t have. They wouldn’t otherwise ask people about the change who aren’t going to have credible input on the topic. Why should they ask the customer about upgrading the backup clients on all the machines when the upgrade process is well-known, well-tested, non-disruptive, and the customer just cares that restores are possible when they need one? It’s nice to tell the customer what’s going on, so they realize that the money they spend on you is actually worth it, but it’s not up to the customer in the end. So don’t make it a choice for the customer (as an aside, I like to call this “sysadmin perogative” — you hired me to make the call so I’m making it).

These system administrators and programmers I speak of were hired to DO things, not to talk about doing things. While there is merit in a planning process when one is warranted, like for changes that cause outages, just because one cowboy doesn’t plan anything doesn’t mean that everybody is like him. Just because another tech likes to hold meetings for many of the changes doesn’t mean that everybody should. This isn’t first grade, and rules intended for one person don’t have to apply to everybody else. Managers, quit frustrating your good guys by saddling them with the B.S. you need for the bad guys. Let them do the jobs you hired them to do.

While I was gone last Friday one of my Linux boxes was DDoS’ed. I’m not sure that the proper response was “oh, cool!” though. Heh, I’ve had nearly every continent scan and attack me but never any DDoS lovin’, and that’s fairly exciting to me. I’m actually a bit annoyed that I missed it. It was all small UDP packets hitting all possible ports, beginning *exactly* at 01:00 CDT and ending *exactly* at 09:00 CDT. Props to the DDoS’ers — they’ve mastered cron. :-) My network colleagues tell me it hit 300,000 flows per hour. Interestingly enough, the Linux box — a single-CPU Dell PowerEdge 2650 running Red Hat Enterprise Linux AS 3 — didn’t seem to notice much, beyond the Gigabit network interface being a little busy.

I’ve been doing some reading on DDoS attacks this evening. Dave Dittrich has some good information over at his web site, and Rob Thomas, though his stuff seems to have gone offline, still has a few presentations out there:

• Rob Thomas’ “Life, Love, and War in the Underground” (PDF)
• Rob Thomas’ “Router Security — Just Add Peers” (PDF)
• Rob Thomas’ presentation on “What Not to Do During an Attack” (PPT) — read the notes
• Dave Dittrich’s web site with lots of links

If I’ve read this right, one of the worst things you can do during a DDoS is freak out and take the host offline. I understand why, but that’s counterintuitive to most people, and I wouldn’t have thought about it during the event itself.

I just spent an extended weekend in New Orleans. That town is amazing. Given the liquor I seem to have consumed, at least based on the rate my wallet was getting lighter (aside from the airport parking kiosk eating my credit card), I can’t even believe the amount of alcohol that must flow through the French Quarter. I was definitely less intoxicated than many people. One neat thing I did was taking the City Segway Tour. While the tour itself wasn’t super informative, the novelty of riding a Segway HT through parts of the French Quarter and Algiers Point far outweighed any negatives. Those Segways are awesome little machines. The controls themselves are fairly simple, once you get used to getting on and getting off. There are three modes a Segway can be in: power assist, balance, and off. Power assist mode is when you are walking beside it, as it just motors along with you. Balance mode is what you ride with. In balance mode leaning forward causes you to move forward, leaning back stops (or goes backwards). Steering is done with a grip on the left handle.

A Segway in balance mode is very much like a canoe – you can be told how to get in and out of one, but until you actually do it you will never learn. The first time you step on to one you have no idea what to expect, and you end up bucking back and forth. If you get on and lean forward you will start moving, and the first time you do that you will overcompensate backwards, and forwards, and back, until you either get off or figure it out. Beyond that, you just have to figure out how not to fly off when you’re turning at speed.

One other novel feature is the keys that start them. Each Segway has three keys, color-coded as black, yellow, and red. The keys are electronic, coded in some way that corresponds to the individual unit as an anti-theft practice. You activate the unit by passing the key across two terminals, completing a circuit. Each key makes the unit function differently, with a different top speed and turning rates. Naturally, the red key is the most dangerous, permitting the top speed of 12.5 MPH and insane rates of turning. The tour started with the black key, moved to the yellow once we were over in Algiers, and then back to black in the French Quarter. It was hard to go back to the black key, as I decided to be slightly irresponsible and chase some pigeons and forgot that you can’t turn as fast with black as you could with yellow. I think the right term is “near miss.”

The tour was supposed to be eight people, but the other four didn’t show up so my three companions and I got to do a lot of excessive messing around on the machines, especially while waiting for the ferry. Now that I know how to ride one, I wish they weren’t so expensive. $5000 seems a bit much for them (even $3000 on eBay is a lot), and it’s no wonder the tour costs $65. Then again, at the rate I was drinking I probably would have consumed $65 worth of beer, anyhow. :-)

Ah, “progress” or a budget finally caught up to the UNIX guys. If these guys hadn’t written UNIX, and the Berkeley guys hadn’t hacked it all up, where would we be today? I have been quite fortunate in that I got a chance to talk to Dennis Ritchie at length a few years ago, in what basically amounted to a chance encounter when I offered to get him a beer at a USENIX evening event. I didn’t even know who he was, just that he needed a drink (the beer was free, I wasn’t being real noble). He mentioned something to me, though, which sticks out in my mind. He said that he was at the right place at the right time, and that any one of us would have done the same thing if we’d been there.

Would we have?

I nominate the following for the sysadmin motto, as stated by God in the “Godfellas” episode of Futurama.

“When you do things right, people won’t be sure you’ve done anything at all.”

It sure seems that way to me. Take, for instance, patching of machines, which is a thankless job because the customer often sees it as an unnecessary inconvenience. If you get hacked, though, it’s blame city. A good customer of mine asked me last Friday if all the downtime to patch is necessary. “Doesn’t it seem sort of reactionary?” he asked. Yes, it is quite reactionary, but I think all this news of Zotob.? and RBOT seems to justify it. I mailed him the link to the CNN article this afternoon. His response: “Thanks for insisting that we patch.” Hey, thanks for listening to me and letting me do my job.

That guy had a few Windows 2000 servers. He also has a firewall protecting him, but it just keeps the real bad guys out. I fear for all of those admins that have Windows 2000 laptops floating in and out of the networks behind the firewalls. Good luck, folks.

To mangle The Clash, I fought the blog and the blog won.

I have avoided a blog for a long time. While trying to avoid blogging I’ve also been yearning to start a system adminstration web site that isn’t just forums. Something with more personality and higher signal-to-noise is in order, where the n00b questions don’t sap the energy of the site. I read a lot of blogs by programmers, such as Phil Haack over at haacked.com and DrunkenBatman at drunkenblog.com, where programming and life as a code-enabled human gets woven into a sort of an ongoing narrative. In hoping to do something similar as a sysadmin, it finally occurred to me that I should do just that: something similar, a.k.a. a blog.

Right now it’s just me yammering on here. Maybe sometime I’ll add others, or do something different. Heck, I haven’t even customized the web template yet, so one thing at a time, I guess. :-) At any rate, thanks for listening, and I hope I make it into your RSS feed reader.