The Lone Sysadmin

Stop patching your machines.

Seriously. You know, patching is overrated. IBM releases an urgent security alert for their OS, and you can just ignore it. They don’t know shit. Red Hat releases a security update, saying that you need to apply it. Screw them, they’re morons. And Solaris would never have a security hole, so why would you need to patch it?

If you want to seem like you’re doing something, like, say, if an auditor is hanging around, subscribe them to the Red Hat Network. It’ll look good, and then you can watch as the number of errata for your systems climbs into the hundreds.

And managers, the single best way to impress me as a tech is to tell me that things like patching machines is taking too much of your group’s time, and it needs to stop. You are clever, and have definitely stumbled upon a deep well of lost staff time. If you do get hacked you get to rebuild all the hacked machines later, but that’s good because it doesn’t take long, and let’s you clean the servers up.

If you’re worried about not patching, remember that a firewall will also keep you completely safe. Firewalls are the only things that effectively block hackers, especially if they run Linux. My suggestion for firewalls is that you find one that is stable, plug it in, and forget it. Don’t bother with a support contract, either — it’s just a waste of money. Updates for firewall firmware (or the OS if it’s Linux) are just going to destabilize the equipment, because that’s how vendors make money. If you resist buying in to their support contract propaganda you’ll have all that extra money to do neat, cool things, and lots of time to do it in.

Phil over at Haacked started this — what are ten blogs you enjoyed reading recently? A timely question, since a friend of mine was just astonished that I read blogs and asked for some recommendations (I haven’t told him I post on one yet — I’m waiting for him to find it sometime on his own). For me it’s been:

Vinography – Alder over at Vinography has some excellent, down-to-earth commentary on wine, with some reviews, some advice. I drink the occassional bottle of wine and love cigars, so his commentary is well received by me. Which also makes me wonder why I haven’t found any cigar blogs yet (haven’t been looking, really).

Fermentations – Tom Wark is a wine PR guy and has some really good commentary on the wine industry. He doesn’t really do reviews, but does talk a lot about bigger-picture sorts of things.

Clublife – Rob over at Clublife is pretty funny. His blog has become really popular, and seemingly burnt out a bit, but the tidbits are still pretty good. You’ll laugh as you go through the archive. At least I did, but as a sysadmin I often harbor similar feelings about my customers as he does of his. I can’t choke my customers out, though.

Opinionistas – The Opinionista is a law student interning at a law firm in NYC. I found her blog via Clublife, and it’s an interesting read if only for the tales of smarmy law firm partners.

Cool Tools – Kevin Kelly’s blog about cool tools is more “tools” in the general sense, rather than hardware. He’s got interesting books, hardware, toys, etc. It’s kinda all over the place, but then again that’s probably why I like it.

Schneier on Security – Bruce Schneier is one of the most outspoken security people around, and his blog is full of interesting tidbits of security-related matter. He’s keen on pointing out the stupidity in the TSA, but that’s what they get when they let a security expert fly.

DrunkenBlog – DrunkenBatman is heavy into the Macintosh programming, and has a cow as his mascot which he encourages readers to photograph in all sorts of places. He’s got more of a community than a blog right now, which I think is cool. I aspire to be that popular.

Gibson Blog – William Gibson’s blog isn’t posted to very regularly (sort of like mine) but it’s usually something a little odd and interesting when it does get attention. I love his books, so maybe that’s why his blog has appeal.

To Philly, From Alaska, w/love – This guy got annoyed with Philadelphia and moved to Tuntutuliak, Alaska to teach. He doesn’t post often but it’s a treat when my aggregator has a story from his blog. He includes a lot of photos, and it’s fun to hear how different things are in more remote parts of the world.

alexking.org: Blog – I first discovered Alex’s blog when I was looking for some software to manage a to-do list for my team (his Tasks Pro software is quite useful in that regard). Since then I’ve been reading his blog. Actually, it’s only because of his WordPress themes contest involvement that I have any theme at all (which I’m still working on). Don’t read his blog on September 19th unless you like pirates.

Update: I almost totally forgot the DirectNIC/Interdictor blog that they’ve been keeping in New Orleans all this time. That’s a twice-a-day read for me, but for some reason I’ve never put it in the aggregator.

Arggh, all themes for Wordpress suck. I’d like one with the sidebar on the left side, and as much space as possible for text on the right. If you’re viewing this site I’m going to have to apologize for the way it’s going to look for the next day or so. I’m on it.

If you’re reading this via RSS, well, you are safe, for now. :-)

Update: I have posted a newer version of this article.

So I just spent about 10 hours of my life getting IPMI working on some Dell PowerEdge 1850s and 2800s so I can cycle their power over the network, and turn them on if the power goes out. That was a lot more challenging than I thought it’d be, mostly because there are about zero good places for someone who generally knows what they’re doing to get an idea of where to start. So here’s my slightly-convoluted guide to configuring IPMI on eighth-generation Dell PowerEdge servers, with emphasis on Red Hat Enterprise Linux AS 4.

This will probably work for Dell PowerEdge 1650s, 2650s, and 1750s, too, but they don’t have a full-fledged BMC. You’ll be able to use IPMItool locally but not over the network. Obviously if you’re using Red Hat Enterprise Linux AS 3 you should grab the packages labeled RHEL3.

You also might want to read Dell’s whitepaper on “Managing Dell PowerEdge Servers using IPMItool.”

Getting it Working

Step 1: Find a server that supports IPMI that is within reach. You’ll soon discover that some of the commands cause bad things to happen, so while you’re learning you’ll want the machine reachable if you put it in an unresponsive state. You’ll also need a machine to send the IPMI commands from.

Step 2: The Baseboard Management Controller (BMC) is piggybacking on the first built-in NIC (eth0, probably) so you’ll have to have that attached to the network.

Step 3: Get the tools you’ll need from Dell. In short, the Dell Dynamic Kernel Module Support utilities are going to be the way to go. The place that has the stuff you need is http://linux.dell.com/files/openipmi/. Grab the latest appropriate openipmi-*-1dkms.tar.gz, and then grab an IPMItool from http://linux.dell.com/files/openipmi/ipmitool/.
You could also get IPMItool from http://ipmitool.sourceforge.net/ and build it yourself.

Step 4: Install the Dell OpenIPMI RPMs by untarring the file and running install.sh. Install IPMItool either by building it or installing the RPM. You will need to install IPMItool on both the target machine and the machine from which you’ll send commands.

Step 5: By default Dell doesn’t enable the IPMI service, so enable it with

/sbin/chkconfig --level 35 ipmi on

Step 6: We have to tell the DKMS autoinstaller, which runs at boot time, to keep these modules updated with our kernels. That means that when you install a new kernel and reboot anything managed by DKMS will get rebuilt. Edit the file /usr/src/openipmi-*/dkms.conf and add a line underneath “PACKAGE_VERSION” that says:

AUTOINSTALL="yes"

I’m not quite sure why they don’t do this automatically. Seems like it’d be handy to me.

Step 7: Unless you can reboot right now we’ll need to start the services manually:

/sbin/service dkms_autoinstaller start
/sbin/service ipmi start

These commands will build the modules for you if your running kernel doesn’t have them already, and then start IPMI. At this point you should have a /dev/ipmi0, which is what ipmitool needs in order to interact with the BMC. It should also persist through a reboot.

Configuring the BMC for Remote Usage

Step 8: The BMC needs an IP address that is in the same subnet/VLAN as whatever you have eth0 (the primary onboard NIC) on. I am assuming that you don’t have 802.1q VLAN tagging on. If you do there the BMC understands that, but I don’t know how to enable that except through the Ctrl-E boot menu at startup. If someone lets me know what to do I’ll add it to this post.

You can set the IP for the BMC via the Ctrl-E boot menu. That might be a good way to start, and you can move on to the command-line stuff later.

Once you have the IP address, gateway, etc. you can issue the commands below. Obviously you’ll need to replace the dummy values with real ones:

/usr/bin/ipmitool -I open lan set 1 ipaddr 192.168.40.88
/usr/bin/ipmitool -I open lan set 1 defgw ipaddr 192.168.40.1
/usr/bin/ipmitool -I open lan set 1 netmask 255.255.255.0
/usr/bin/ipmitool -I open lan set 1 access on

Step 9: Now we need to secure the BMC so that random people can’t hose your machines. Do this even if your network is secure. It’s the whole defense-in-depth thing, where you have multiple layers of protection. Plus, you don’t want some n00b admin discovering IPMI and inadvertently rebooting your machines, do you?

I use mkpasswd, which comes with Expect. I love that thing. The command:

/usr/bin/mkpasswd -s 0

will give you a nice eight character random string, devoid of special characters. which you can feed into lots of stuff using backticks.

You have two passwords and one SNMP community string to set. The BMC has a null user, and a user named “root.” One of the passwords, the root user’s, will be the password you use to interact with the BMC via the network.

This sets the SNMP community. Replace YOURSNMPCOMMUNITY with something else, like a string from mkpasswd. Unless, of course, you plan to use the SNMP functionality. I’m not that advanced, yet, and ipmitool does everything I need, so you’re on your own with that:

/usr/bin/ipmitool -I open lan set 1 snmp YOURSNMPCOMMUNITY

This sets the null user’s password. Replace CRAPRANDOMSTRING with something else:

/usr/bin/ipmitool -I open lan set 1 password CRAPRANDOMSTRING

This sets the root user’s password. Whatever you set this to, remember it:

/usr/bin/ipmitool -I open user set password 2 REMEMBERTHIS

Yay!

Double-check your settings with:

/usr/bin/ipmitool -I open lan print 1

Doing Stuff to Your Machine from Afar

Step 10: Now you get to try messing with the power remotely. Go to the machine that just has IPMItool on it and issue the command:

/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power status

Obviously you’ll need to use the IP you set the BMC to in Step 8. After entering the password you used for the root user in Step 9, you should see:

Chassis Power is on

Unless the power is actually off, but at this point I’m trusting that you’re not that clueless. :-)

If you get anything else, or nothing, double-check to make sure the BMC is set right, you entered the right password, and the IP it has is reachable from the machine you’re on. You can double-check your work via the Ctrl-E boot menu, too.

From here, the sky is the limit. During your testing make sure you’re familiar with:

/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power off

/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power cycle

There are a lot more commands you can issue via ipmitool, so read the man page or the help information.

Let me know if something here is wrong. After 10 hours of messing with this I’m a little fried. :-)

William Gibson has a wonderful entry on his blog from the other day, regarding old Dutch names like Crapahildis. It is no wonder why names like that, and Hortense, etc. have died out. My god.

It’s strange, though, that I feel a machine naming scheme coming on…

Okay, so the blog has been suffering a little. Sorry folks. I’ve just been burnt out. My primary job has been hectic lately, as management reorganized quite a large chunk of the system administration and “base technologist” jobs a month ago. I’m not complaining at all, actually, even though the last two weeks have been 70 hour weeks. I know I’m not going to get any sympathy when I say that I really try to keep to 40 hours a week (hey, work smarter not harder). Sometimes a reorg can be a good thing, if you are able to get some things fixed in the process.

One thing that is really getting fixed are some of the operating system teams. We’re not a one vendor shop. We run IBM AIX, Sun Solaris, Red Hat Enterprise Linux, Microsoft Windows, Apple Mac OS X, and an IBM z/OS mainframe. There is a team tasked to the care and feeding of each OS. The problem is that some teams were in pretty good shape, but some really sucked.

I read “The Trouble With Open Source” and while Stephen Marshall is talking about software development, he puts his finger on something system administrators absolutely need to succeed: a singular vision and clear design concept.

How can a system administrator have a vision when they work on machines for so many projects? A clear design concept? Heck yeah. An admin with a vision won’t have the vision at the application level (unless they’re implementing it). They’ll have it at the OS level, with standardization and forethought being big symptoms of vision. Your admins will be thinking ahead to what their customers might need, and what would make their users, and their own, lives easier.

How can you tell when there’s a lack of vision, and a lack of a clear design concept? Instead of a guiding voice there is infighting. Instead of leaving a meeting knowing what needs to be done they leave a meeting griping about their manager making a decision they didn’t like. Instead of welcoming help from others when they’re busy they resist any involvement by others in their projects. It’s my project. Those are my machines. Look how many machines I have compared to the other admins! I want a raise. No, I’m sorry, we can’t install that Perl module for you because it needs a newer version of Perl than the one that shipped with Solaris 8. No, we don’t install newer versions of Perl. Why? Because I said so.

Fixing this is really easy. Every team needs a leader. Not to take over the job of management, but to make technical decisions, provide guidance, and to drive the ruthless standardization every team should have. The leader is a technical leader who provides feedback and advice to management as to what should be worked on by the team, but doesn’t make personnel decisions or work assignments. If there’s a disagreement about how something technical should happen the team discusses it, the technical lead makes a decision, and management agrees. They’re a technical leader, not a team manager.

So, when I say that a reorganization can be good if things get fixed, I’m thinking about the restructuring of our OS teams. Each team has a leader now, backed by management, and charged with making some sense of the mess the old setup left. In the last three weeks infighting is down to negligble levels and the teams are starting to get things done that should have been done months ago. There’s even a technical lead above all of the team technical leaders, charged with standardizing common cross-platform tasks so customers will have the same experience regardless of whether they ask the Linux team or the Solaris team.

Time will tell if this works or not, but the initial signs, as a Magic 8 Ball would say, “point to yes.”

Ah, New York City is a wonderful place. But it’s good to be home now, too. More later, when I get a chance to sort through all my email.