Update: I have posted a newer version of this article.
So I just spent about 10 hours of my life getting IPMI working on some Dell PowerEdge 1850s and 2800s so I can cycle their power over the network, and turn them on if the power goes out. That was a lot more challenging than I thought it’d be, mostly because there are about zero good places for someone who generally knows what they’re doing to get an idea of where to start. So here’s my slightly-convoluted guide to configuring IPMI on eighth-generation Dell PowerEdge servers, with emphasis on Red Hat Enterprise Linux AS 4.
This will probably work for Dell PowerEdge 1650s, 2650s, and 1750s, too, but they don’t have a full-fledged BMC. You’ll be able to use IPMItool locally but not over the network. Obviously if you’re using Red Hat Enterprise Linux AS 3 you should grab the packages labeled RHEL3.
You also might want to read Dell’s whitepaper on “Managing Dell PowerEdge Servers using IPMItool.”
Getting it Working
Step 1: Find a server that supports IPMI that is within reach. You’ll soon discover that some of the commands cause bad things to happen, so while you’re learning you’ll want the machine reachable if you put it in an unresponsive state. You’ll also need a machine to send the IPMI commands from.
Step 2: The Baseboard Management Controller (BMC) is piggybacking on the first built-in NIC (eth0, probably) so you’ll have to have that attached to the network.
Step 3: Get the tools you’ll need from Dell. In short, the Dell Dynamic Kernel Module Support utilities are going to be the way to go. The place that has the stuff you need is http://linux.dell.com/files/openipmi/. Grab the latest appropriate openipmi-*-1dkms.tar.gz, and then grab an IPMItool from http://linux.dell.com/files/openipmi/ipmitool/.
You could also get IPMItool from http://ipmitool.sourceforge.net/ and build it yourself.
Step 4: Install the Dell OpenIPMI RPMs by untarring the file and running install.sh. Install IPMItool either by building it or installing the RPM. You will need to install IPMItool on both the target machine and the machine from which you’ll send commands.
Step 5: By default Dell doesn’t enable the IPMI service, so enable it with
/sbin/chkconfig --level 35 ipmi on
Step 6: We have to tell the DKMS autoinstaller, which runs at boot time, to keep these modules updated with our kernels. That means that when you install a new kernel and reboot anything managed by DKMS will get rebuilt. Edit the file /usr/src/openipmi-*/dkms.conf and add a line underneath “PACKAGE_VERSION” that says:
I’m not quite sure why they don’t do this automatically. Seems like it’d be handy to me.
Step 7: Unless you can reboot right now we’ll need to start the services manually:
/sbin/service dkms_autoinstaller start
/sbin/service ipmi start
These commands will build the modules for you if your running kernel doesn’t have them already, and then start IPMI. At this point you should have a
/dev/ipmi0, which is what ipmitool needs in order to interact with the BMC. It should also persist through a reboot.
Configuring the BMC for Remote Usage
Step 8: The BMC needs an IP address that is in the same subnet/VLAN as whatever you have eth0 (the primary onboard NIC) on. I am assuming that you don’t have 802.1q VLAN tagging on. If you do there the BMC understands that, but I don’t know how to enable that except through the Ctrl-E boot menu at startup. If someone lets me know what to do I’ll add it to this post.
You can set the IP for the BMC via the Ctrl-E boot menu. That might be a good way to start, and you can move on to the command-line stuff later.
Once you have the IP address, gateway, etc. you can issue the commands below. Obviously you’ll need to replace the dummy values with real ones:
/usr/bin/ipmitool -I open lan set 1 ipaddr 192.168.40.88
/usr/bin/ipmitool -I open lan set 1 defgw ipaddr 192.168.40.1
/usr/bin/ipmitool -I open lan set 1 netmask 255.255.255.0
/usr/bin/ipmitool -I open lan set 1 access on
Step 9: Now we need to secure the BMC so that random people can’t hose your machines. Do this even if your network is secure. It’s the whole defense-in-depth thing, where you have multiple layers of protection. Plus, you don’t want some n00b admin discovering IPMI and inadvertently rebooting your machines, do you?
I use mkpasswd, which comes with Expect. I love that thing. The command:
/usr/bin/mkpasswd -s 0
will give you a nice eight character random string, devoid of special characters. which you can feed into lots of stuff using backticks.
You have two passwords and one SNMP community string to set. The BMC has a null user, and a user named “root.” One of the passwords, the root user’s, will be the password you use to interact with the BMC via the network.
This sets the SNMP community. Replace YOURSNMPCOMMUNITY with something else, like a string from mkpasswd. Unless, of course, you plan to use the SNMP functionality. I’m not that advanced, yet, and ipmitool does everything I need, so you’re on your own with that:
/usr/bin/ipmitool -I open lan set 1 snmp YOURSNMPCOMMUNITY
This sets the null user’s password. Replace CRAPRANDOMSTRING with something else:
/usr/bin/ipmitool -I open lan set 1 password CRAPRANDOMSTRING
This sets the root user’s password. Whatever you set this to, remember it:
/usr/bin/ipmitool -I open user set password 2 REMEMBERTHIS
Double-check your settings with:
/usr/bin/ipmitool -I open lan print 1
Doing Stuff to Your Machine from Afar
Step 10: Now you get to try messing with the power remotely. Go to the machine that just has IPMItool on it and issue the command:
/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power status
Obviously you’ll need to use the IP you set the BMC to in Step 8. After entering the password you used for the root user in Step 9, you should see:
Chassis Power is on
Unless the power is actually off, but at this point I’m trusting that you’re not that clueless. :-)
If you get anything else, or nothing, double-check to make sure the BMC is set right, you entered the right password, and the IP it has is reachable from the machine you’re on. You can double-check your work via the Ctrl-E boot menu, too.
From here, the sky is the limit. During your testing make sure you’re familiar with:
/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power off
/usr/bin/ipmitool -I lan -U root -H 192.168.40.88 -a chassis power cycle
There are a lot more commands you can issue via ipmitool, so read the man page or the help information.
Let me know if something here is wrong. After 10 hours of messing with this I’m a little fried. :-)
Comments on this entry are closed.
Thanks for that. Interestingly on my fairly stock RHEL4 installation trying to hit the IP assigned to BMC from the machine itself doesn’t work yet remotely it does.
For some reason, the graceful shutdown feature works perfectly on RHEL servers (using IPMI), but on my Windows 2003 servers, it does not. Is there some W2K3 config I need to make to allow BMC shutdown?
For the vlan (802.1q) question i would guess that it must be set using the following command (Dell 1950, openipmi 2.0.2):
ipmitool lan set 1 vlan id
(Be carefull while testing i wasn’t able to switch off
Thanks for ur hints. Now IPMI is working well. but did u figure out how to read things via snmp?
would ease some processes..
I would add that because the BMC has it’s own TCP stack, it makes good sense to use a different fakenet IP on the same subnet. For obvious reasons the local system still won’t be able to ping that IP but of course remote machines will.
Thanks for getting this together. I was able to get things running quickly. The world needs more “I’m not a raving moron” guides just like this.
I’m glad I found this post, I am still having lots of trouble with Remote access to the BMC on a Poweredge 830. Should I be able to ping the BMC IP address from another machine? I cannot. My arp table does show the correct IP to MAC association and that the address is reachable. All remote ipmitool commands fail.
Any suggestions? Thanks!
This is great info and appreciate you sharing w/ the computing community at large. I have 35 boxes to get this running on and all of them are different models. Your tutorial (or exercise in frustration) will prove useful in getting to the end result much more efficiently. Thank you. :)
Last night I was using ipmi on our cluster of PowerEdge SC1425’s (stripped down 1850’s, but they still have the BMC). Once the power was off, ipmitool could not talk to the BMC. Once I powered the cluster back up, ipmitool worked fine. Ever see it where the BMC was not reachable in the power-off state?
The server draws its power from the mains (obviously). As long as the mains cord is connected the BMC will be active.
If the shutdown process on your server “physically” downs the ethernet interface then the interface is no longer visible to the rest of the network (check your switch, you’ll see the port down). You need to ensure that your shutdown process doesn’t do an “ifconfig eth0 down” (or similar). Many (Linux) distributions now have hooks for this in their rc scripts or the relevant config files.
Thanks! Works perfect! (even on IBM x3550)