With all the CPU and other hardware vulnerabilities present in modern servers it’s very important to keep your Dell firmware updated. Various operating systems, like VMware ESXi, Linux, and Microsoft Windows ship CPU microcode with their OSes, but that only updates the CPUs. There are still other vulnerabilities that exist, like in the Intel Management Engines, memory controllers, UEFI firmware, and so on that if left unpatched present an opportunity for attackers.
I dislike complexity. A lot. Adding more tools to an environment in order to do a job, like managing Dell firmware updates on servers, also adds complexity. Thankfully, on Dell PowerEdge servers there are a couple of approaches to keeping firmware updated that don’t add a lot of overhead. In short, they mostly involve getting the iDRAC to stage updates, so that when the system boots, the Unified Server Configurator or Dell Lifecycle Controller will apply them automatically.
First: Clear Old iDRAC Job Data
The first step before you try downloading and staging firmware updates to an iDRAC is to clear the old job queue. By doing this you will be able to easily tell if your job succeeded or failed. You can do this in the iDRAC UI, which is fine if you only have a few hosts, or you can use the racadm/racadm.exe command line utilities to help automate the process:
racadm.exe -r <iDRAC_FQDN_OR_IP> -u root -p '<PASSWORD>' jobqueue delete --all
Give it a few minutes to complete; embedded controllers like iDRACs and iLOs aren’t that fast.
Option 1: iDRAC Direct Firmware Downloads
The first option is the simplest: tell the iDRAC to download and stage the updates directly from downloads.dell.com. This requires that your iDRACs have outbound access to that site, of course. If they do, all you have to do is specify downloads.dell.com as the HTTPS URL in the iDRAC update tab, or use the racadm/racadm.exe command line utilities:
racadm.exe -r <iDRAC_FQDN_OR_IP> -u root -p '<PASSWORD>' update -t https -e downloads.dell.com -a FALSE
The “-a FALSE” at the end tells it to not restart the machine, just stage the updates and wait. This works well with things like VMware vSphere Lifecycle Manager and Microsoft Windows Update, as they patch and restart the servers the servers will update themselves.
You can also execute these commands from the SSH interface of the iDRAC. Should you have iDRAC SSH on? That’s up to you, but securing your iDRAC is an important step.
Once you issue these commands you should see jobs appear in the iDRAC job queue as scheduled:
Once you restart they’ll say completed or failed.
Option 2: Use an Internal Repository
There are three main problems with Option 1. First, if you have a lot of hosts, Dell might release new firmware in the middle of your updates, and now you have servers on two different versions. Uncool. Second, you have to let your iDRACs talk to the world, which is probably not the greatest plan. Third, the iDRACs all going out and downloading things will be a lot of network traffic.
Dell has a tool called the “Dell Repository Manager” that lets you build offline repositories for use with iDRACs and other OpenManage system tools. You can specify the server models you have and it’ll create a bundle that you can put up on a web server inside your organization. Then you can use Option 1, but specify that host instead of downloads.dell.com.
Frankly it’s a heck of a lot easier to create a small Windows or Linux web server to host a repo like this than it is to use other proprietary tools and have to maintain them. I use a Rocky Linux VM for it, running httpd, and now that Windows has ssh & scp natively built in it’s super easy to copy the exported repository to the Linux web server. Make a directory in /var/www/html, like “idrac-repo-20240507” in my example, set it to be owned by your user (chown -R plankers /var/www/html/idrac-repo-20240507), and then use scp to copy from your desktop:
scp -r exported_repo_dir [email protected]:/var/www/html/idrac-repo-20240507
Tips & Tricks for Dell Firmware Updates
If you’re having issues with downloads, make sure you’ve configured NTP on your iDRACs. TLS connections need the time to be relatively accurate. Plus, you should be sending syslog information from the iDRAC to your SIEM anyhow, and that needs accurate timestamps to be useful.
Do one host first and watch it, so that any unintended side effects of an update get caught early. You might consider using the GUI on a host to see what updates are available for your fleet.
Don’t stage firmware a long time before you plan to reboot. If the host crashes or you have a power outage the hosts will update when they restart, and that’s a complication you may not want in the middle of another incident. Though it might be an opportune time to actually get the updates done… don’t let a crisis go to waste.
If there are updates to the iDRACs themselves they’ll auto-apply, regardless of the “-a FALSE” setting. This generally causes a brief iDRAC outage and some weirdness, though it’s not usually a problem if you know to expect it.
If you do need to cancel the update jobs just use the command above for clearing the job queue.
As always, good luck.
