The Lone Sysadmin » System Administration

Three Organizational Decisions That Help Me Virtualize

Bob Plankers — Fri, 06 Aug 2010 18:55:55 +0000

Over the last ten years my organization has come a long way with its IT policies and processes. We’ve gone from the wild, wild west of IT where personal heroism ruled the day, to a place where there’s just enough process to make sure that communication happens correctly and things like our Configuration Management Database (CMDB) stay up to date. It’s been a lot of work, but I am actually really proud of where we’re at.

There are three fundamental decisions we made a long time ago that, had they not been made, would have drastically changed how virtualization has proceeded here.

1. Clearly defined maintenance windows.

Knowing exactly when someone can do maintenance on server has been crucial to getting things done in our virtualization environment. There are many adjustments you can & should make in virtual environments, but if you can’t ever take the VMs down to make the changes you’re stuck. We’ve been able to do physical to virtual migrations, performance tuning, VMware Tools upgrades, vSphere upgrades, and a whole slew of other things in relatively short timeframes because we have this all worked out already. This also lets us “right-size” our VMs — rather than deploying huge VMs just in case they need the CPU or RAM, we deploy smaller ones and then can take an outage to add CPUs and RAM if we need to. The maintenance windows for a server are negotiated between the application/service admins and the system administrators when a machine is put into production, we track it in our CMDB, and any member of the whole team supporting the service can take the maintenance window, as long as they follow some rules about notifications for the change (timeframes, etc.).

2. Use of load-balancing technologies.

We use application load balancers (layer 4 of the OSI model) to decouple services from individual servers. Not only does this allow us to take a host down without affecting a service, but it also lets us spread the load out more among the physical hosts we have in our virtual infrastructure. In a lot of cases having more, smaller VMs results in better workload scheduling by ESX and DRS, especially on smaller ESX hosts.

Of course, this also plays nicely into the other points, because it’s very liberating to be able to do what we call “rolling maintenance” on a service, just taking one machine down at a time so that customers are not impacted. It also means that system administrator quality of life goes up, for now we can do maintenance tasks during the day instead of on weekends and off-hours. Doing maintenance during business hours has a couple of benefits. First, it means that the maintenance will actually get done. If you try to use someone’s personal time to do work they tend to opt out of that work. Servers go unpatched, tuning doesn’t happen, lots of things that should get done don’t because people will choose their personal time over work. Second, it means that if something goes wrong there are others around to help out. Doing work at 5 AM on a Sunday is fun, but if things go sideways you have to wake someone up or try fixing it yourself. Doing work during the day means you have the rest of the team around to lend a hand.

Third, it gives you a way to make incremental changes and then watch the effects. This has been particularly awesome for performance tuning of applications and our virtual environments themselves. Testing tuning changes is often hard, because test suites and test load generators are synthetic and often don’t compare to real load. But because the load is spread out we can make a change to one VM, or one ESX host servicing one VM, and keep an eye on it. I’m not advocating being a complete cowboy — you still have to do testing — but the risks to your production environment are a lot lower if you can catch problems on one VM first.

There are usually some other benefits to load balancers, too, that make them virtualization-friendly. Many will offload SSL processing, so your VMs have less work to do. Others have features, like iRules in F5′s products, that let you rewrite network traffic on the fly, which has some really neat implications for security, monitoring, and service delivery. And if you don’t want to buy a piece of hardware you can often get a virtual appliance from these vendors, though the physical appliances are usually a lot faster.

3. Commitment to operating system and application patching.

It is a fundamental belief of mine that one of the best ways to stay secure is to keep up on your patching. My organization agrees, and by using load balancers and defining maintenance windows we’ve made it easy for ourselves to keep our hosts up to date with regular patching cycles. Because we can take servers down without taking services down, and because sysadmins know exactly when a server can come down, we can schedule maintenance cycles easily, whether it’s six months out or two weeks. We can also respond very rapidly to emergency situations, like recent remote execution vulnerabilities in Microsoft Windows, by rolling patches out to development & test hosts, then QA & production, over the course of just two days if needed.

Keeping up to date with patches not only keeps you secure, it also lets you take advantage of new features that are added to operating systems. For example, Red Hat keeps adding new virtualization-friendly features, like kernel interrupt clock dividers. Being a kernel parameter you can’t just change it on the fly. And if you have to reboot, but can’t get a time to do it, you won’t do it. For us, we just rolled the change into one of our patching cycles and reduced the load on our infrastructure dramatically. Meaning more VMs per physical host, and a quantifiable amount of savings from just a small change on each machine.

Furthermore, our commitment to patching also extends to the virtual infrastructure itself, and we have a rule that we will not implement anything that breaks vMotion or Storage vMotion. Why? Because then it becomes very difficult to cope with ESX updates, or hardware failures, or any situation where vMotion could be used to prevent an outage. Sure, this means that we still need physical hardware for some applications, but it’s still just a fraction of the hardware we were buying years ago. This also makes virtual infrastructure easy to upgrade when the time comes, for new versions of vSphere, new storage arrays, and new physical hosts. Instead of planning outages on hundreds of VMs we just vMotion them, and nobody is the wiser.

Disclosure: F5 is a sponsor of Gestalt IT Tech Field Day, of which I have been a participant. I am not a customer of F5 at this time, though.

Happy System Administrator Appreciation Day!

Bob Plankers — Fri, 30 Jul 2010 17:01:06 +0000

The Wisconsin DMV sent me my gift a day early:

And it was a present — I needed replacement plates but hadn’t ordered them yet. I’m glad I didn’t!

I often joke that I haven’t come up with an original solution to anything in years, thanks to all the other sysadmins out there who share their solutions, knowledge, and time in order to make the world better. Thank you all for everything you do!

Rate-Limiting Steps

Bob Plankers — Mon, 26 Jul 2010 18:02:14 +0000

In the last month I’ve added quite a few blogs to my reading list. One new one is “Movin’ Meat,” written by an ER doctor out of the Pacific Northwest. Besides just being interesting, some of his blog posts support my theory that IT folks can often learn things from people in other fields. The post from June 25, 2010, part four of his “Advice for Interns,” is one of these cases. When you read it (link is at the end because I want to get to my actual point before you leave to read it), I think substituting “customer/system” for “patient” in his list works nicely.

My real point is this: one thing in his list really stood out for me. It’s something that seems really obvious when it’s said, but also done wrong a lot:

“Determine the rate-limiting step and make it priority #1 in the work-up”

Figuring out what the slowest step in a project is going to be and getting to work on it right away is often key to getting a project done in a timely fashion. Especially if a large chunk of that time will be waiting for something. When you know it’s going to take six weeks for a request to make it through your purchasing department you should start that right away, especially since all you’ll be doing is waiting.

As kids we were told by our teachers to read all the instructions first, then start working on whatever we were doing. Determining the rate-limiting steps is the same sort of thing. By taking a few minutes at the beginning to look at the whole project first, rather than just starting on step #1 and going one by one until you’re done, you can often optimize things so that the longest parts of the project are done in parallel with the rest.

Links:

- Movin’ Meat: Friday Flashback – Advice for Interns Part Four

Why No-Reply Email Is A Bad Idea

Bob Plankers — Thu, 08 Jul 2010 21:17:32 +0000

I absolutely hate no-reply email. I understand why it exists (autoresponders and bounces), but to send an email with no way to respond at all using the same communications medium is ridiculous.

A good example of this is the customer satisfaction survey Red Hat just sent me. It is from a no-reply email address and there is no other email address listed. There is just some text and a URL, and clicking on the URL gets me:

$ dig rhapps.redhat.com ns1.redhat.com
[...snip...]
;; QUESTION SECTION:
;rhapps.redhat.com.             IN      A

A records are overrated.

I generally am a nice guy and let vendors know something is messed up, but there are limits, especially when I’m already on the fence about a negative experience. I’m not going to open a support case with them, because it’ll never get escalated correctly[0]. And there’s no email address to send a quick note to. So it goes unfixed, Red Hat gets added to my mental list of vendors that don’t get it, and I blog about it, which may be worse than a negative survey response.

And for heaven’s sake, if you send out a customer satisfaction survey make sure it works.[1]

——————–

[0] Here’s a test for your organization: can a customer open a support problem against your web site? Will it go to the right place, i.e. the webmasters or someone intelligent who can get things fixed? If not, why not?

[1] This also may mean you should add an external survey service to what you monitor.

More Equipment Means More To Go Wrong

Bob Plankers — Wed, 30 Jun 2010 17:04:47 +0000

“Everything should be made as simple as possible, but not simpler.” – Albert Einstein

Over the last couple years my organization has been building out an alternate site, for disaster recovery and business continuity purposes. I’ve noticed a disturbing trend, both among my own coworkers and others who are starting to think about DR & BC: that by having multiple data centers, multiple servers, or using multiple cloud vendors they’ll reduce the number of problems they’ll have.

From a system administrator’s point of view that idea is absolutely false.

Every piece of equipment you have can fail, and the more equipment you possess the more likely a failure will be. The more servers, switches, chillers, PDUs, and power grids you have the more likely it will be that one dies. Heck, that’s even true of components within the servers. As I add more servers to my own environment, each with mirrored disks and tens of DIMMs, I spend more time replacing failed drives, failed DIMMs, updating firmware, etc.

Alternate data centers also add other problems than just mechanical failure. Your second data center is probably attached to a different power grid, so now you have two power utilities to worry about, and are subject to twice as many outages. You also have to maintain the equipment in that other data center, change the air filters on the air handlers, etc. And it isn’t twice as much work as having one data center, it’s three or four times when you count the overhead of traveling to a remote site, not having tools available, etc.

Having alternate sites, and multiple servers, can certainly improve application reliability by keeping services available to customers in case something happens. It’s a good bet that a power problem in San Jose won’t affect your site in Omaha, for example. But it absolutely increases your odds of having equipment failure, and while your customers might be happy your operations folks won’t be.

More equipment means more that can go wrong. So what do you do? Keep it simple. Have fewer, bigger machines. Have machines with fewer moving parts. Use a cloud provider with multiple sites. Use VMware DPM and keep your alternate site machines in standby most of the time, and the HVAC off. There are lots of ideas, but remember that the more you have, the more that will fail, and those failures mean time spent not moving forward.

Levels of Indirection

Bob Plankers — Tue, 29 Jun 2010 20:08:21 +0000

“All problems in computer science can be solved by another level of indirection…
Except for the problem of too many layers of indirection.”

- David Wheeler, though often attributed to Butler Lampson, who has some great quotes, too:

“When in doubt, use brute force.”
“In handling resources, strive to avoid disaster rather than to attain an optimum.”

Lots of good stuff if you read his “Hints for Computer System Design.”

Midnight is Always Tomorrow

Bob Plankers — Mon, 14 Jun 2010 17:17:22 +0000

“So, are you ready for the big power outage on Sunday?” a colleague asks on Thursday.

“You mean Saturday.”

“No… Sunday morning.”

“Um, I was told two months ago, and countless times between, that the outage is on Saturday, midnight to 8 AM, and they were starting to shut things down at 10 PM.”

“It’s Sunday, midnight to 8 AM. They’re going to start shutting things down on Saturday at 10 PM.”

“Did they move the outage?”

“No, I bet they were just telling you when things were going to start. On Saturday.”

Midnight is 00:00, meaning the start of a new day. Always.

If you’re in doubt, use 00:01. Assume everybody is clueless about time, because they are. For example, a lot of people think in terms of when they go to sleep, not what actual time it is, so if they’re still up at 0200 on Sunday they consider it to be Saturday. While that’s wrong, and makes visions of their painful, torturous death flash in your mind, it’s a fact of life. Deal with it.

Be precise. Use 24-hour time, because there is no AM/PM question. 24 hour time runs between 0000 and 2359 on any given day. There is no 2400[0].

Last, all times should be accompanied by days, and vice-versa. It’s like units in science classes. You didn’t just write “1.67,” you wrote “1.67 meters.” It isn’t “0800,” it is always “0800 on 4/18/2010.” Times are useless without dates. And if your team or customers are not all in the same time zone, and they rarely are[1], you need that information, too.

“The system shutdowns will commence at 2200 on 4/17/2010, the power will be disconnected at 0000 on 4/18/2010, and power-ups will occur again at 0800 on 4/18/2010. All times are in CDT (-0500).”

———————

[0] Yes, I am aware there are sometimes leap seconds, which get added to the end of a day, thus causing a 23:59:60. 99.99%+ of all outage planning does not need to take this into account.

[1] And even if they are, it doesn’t hurt to add that information.

What are P-states and how do I use them in vSphere?

Bob Plankers — Fri, 21 May 2010 18:08:34 +0000

VMware vSphere 4 added the ability to take advantage of Intel SpeedStep and AMD PowerNow! CPU power management features. These features are commonly known as “Dynamic Voltage and Frequency Scaling” or DVFS, and let an OS cooperate with the CPU to reduce power consumption by reducing the frequency of the CPU and the voltage at which it is operating. It reduces these things in preset tiers, and these tiers are known as P-states. On Intel CPUs they are trademarked as “SpeedStep” and on AMD they are either “Cool’n'Quiet” or “PowerNow!”

The Wikipedia article on Intel SpeedStep points out that “power consumed by a CPU with a capacitance of C, running at voltage V, and frequency f is approximately P = CV²f.” This means if you can reduce the voltage to the CPU the power needs drop in a non-linear fashion. Furthermore, many electronic components run more efficiently at lower temperatures, and since consuming less power means less heat generated you end up seeing efficiency gains within the host as well as reduced load on data center cooling. This results in an overall reduced power bill, and potential savings in related systems like a UPS, generators, etc.

Frequency and voltage in a CPU are correlated. So are instructions per second and frequency. Basically, if you want your CPU to get more work done per second you need to increase the frequency it runs at, and to do that you need to increase the voltage. So why would you want to turn the CPU’s performance down in the first place? The thing is, CPUs are much faster than everything else in a computer system. If the CPU needs data for an operation it’ll look in cache. L1 cache operates at the CPU speed — fast but small. L2 cache operates at a fraction of the CPU speed, but still many times faster than RAM[1]. The problem is when the CPU needs data that isn’t found in cache and has to go to RAM or disk. Going to RAM means it’ll wait for thousands of clock cycles before the data is returned, because RAM is much slower than the CPU. Going to disk or network means waiting for millions of clock cycles, which is an eternity to a CPU. So while the system may be busy, the CPU might actually be idle, and that’s a great time to stop using power and generating heat.

When one process is doing I/O like that it’s also a good time for the hypervisor in vSphere (or scheduler in a regular OS) to run something else. That “something else” might not need the full performance of the CPU, either, and the frequency & voltage of the CPU can be decreased to save power in that case, too.

Given that all this trouble has been taken to add this feature to hardware and software, how do you turn it on?

1. Make sure your CPUs have this feature. According to VMware vCenter, under Configuration->Processors, my sample Dell PowerEdge R610 has Intel E5530 CPUs. I can check that by looking at Intel’s product web site, ark.intel.com, under “Xeon” processors.

2. If, in vCenter, under Configuration->Processors it has something like “Enhanced Intel SpeedStep” listed by “Power Management Technology” then you can proceed to step 3. If it says “Not Available” or something else you may need to set your BIOS to allow operating system control of the power management. On my Dell PowerEdge R610 the option is under Power Management. Set it to “OS Control” as:

On some older models, like the PowerEdge R900, it’s in the CPU options and called “Demand-Based Power Management.”

3. Go back in vCenter. By now the Power Management Technology should be populated with something other than “Not Available” (if that isn’t the case then check with your hardware vendor). If that’s set, go to Configuration->Advanced Settings, then Power, and change Power.CpuPolicy to “dynamic.”

4. Say OK and you’re set.

I’ve added this to my checklist for bringing a new ESX host online now, and now that I’ve got it enabled I’m watching the power consumption a lot more closely. Can I tell a difference? Hard to say right now, as I don’t have enough new data for my small clusters. It still doesn’t replace Dynamic Power Management (DPM), because if you genuinely don’t need the capacity of a host shutting it completely off makes the most sense. But in the effort to be greener, every little bit helps, and it’s easy to enable.

As always, if I’ve made a mistake or you’d like to add relevant information just make a comment below. I read all my comments!

——————–

[1] This is why larger L1 & L2 caches are better, why prefetchers exist (to try prepopulating the caches with data the CPU might need), why architectures like Intel’s Nehalem add L3 caches that are shared among the cores, and why hypervisors try to schedule the same process on the same CPUs when they can (CPU affinity increases the chance that useful data is still in the caches). It’s all a big effort to keep the CPUs from waiting.

If You Don’t Like Change…

Bob Plankers — Thu, 20 May 2010 19:32:40 +0000

“If you don’t like change, you’re going to like irrelevance even less.” – General Eric Shinseki, former United States Army Chief of Staff.

Power Consumption of a Dell PowerEdge R610

Bob Plankers — Thu, 13 May 2010 16:44:14 +0000

For planning purposes I just did some power draw testing of a Dell PowerEdge R610. Dual Intel X5550 CPUs, 24 GB of RAM, four SSD disks attached to the PERC6/i, and dual 717 Watt power supplies. My testing methodology was to measure the draw using a Fluke 322 clamp meter, both at idle and running a stress test under Red Hat Enterprise Linux 5 (stress -c 32 -d 8 -i 8 -m 16). I did this with one and two power supplies active.

1 PS, idle: 0.65 Amps @ 202.3 Volts = 131.5 Watts

1 PS, loaded: 1.51 Amps @ 202.3 Volts = 305.5 Watts

2 PS, idle: 0.35 Amps @ 202.3 Volts = 70.8 Watts each (total of 141.6 Watts)

2 PS, loaded: 0.77 Amps @ 202.3 Volts = 155.8 Watts each (total of 311.6 Watts)

Virtualization users who didn’t see the VCritical commentary on “Idle RHEL Hypervisors save power?” might want to check that out, since these numbers directly support Mr. Gray’s argument. An idle server drawing 45% of the power of a loaded server is a pretty solid argument for VMware DPM.

Also note that the label on the power supplies indicates these are 717 watt power supplies, while the maximum draw I recorded was less than half that. While building in some overhead is a good idea, using the labels to determine draw isn’t a great idea, because you will overbuild your infrastructure.

Update: As was pointed out in the comments, you can use the Dell iDRAC web interface to find out the current power consumption of the 11th generation Dells. However, it looks like that has accuracy problems, particularly at the low end (idle), but is relatively usable at the top end. If you’re serious about measuring this stuff I’d still get a meter, though.

Dell PowerEdge R610 & PERC/6i Disk Comparison

Bob Plankers — Tue, 30 Mar 2010 19:48:46 +0000

I’ve recently done some very basic disk performance testing of a Dell PowerEdge R610 with 24 GB of RAM (1333 MHz), dual Intel X5550 CPUs, a PERC/6i RAID controller, and a bunch of 146 GB 15K RPM 2.5″ disks, as well as four of the Dell 50 GB enterprise SSD disks (which are Samsung drives). I tested various combinations of RAID 0, 1, 5, 6, 10, and 50 with 1, 2, 3, 4, and 6 disks.

While the RAID controller configurations varied, all the configs had the element size set to 64 KB, read policy set to Adaptive Read Ahead, and write policy set to Write Back. The PERC/6i firmware was 6.2.0-013. The operating system was Red Hat Enterprise Linux 5 Update 4, 64-bit, updated to the latest patches as of 3/10/2010. The filesystems were all LVM-based ext3 filesystems, formatted with “mke2fs -j -m 0 -O dir_index.” I used the benchmarking command bonnie++, in the form “bonnie++ -r 32768″ to indicate that I had 32 GB of RAM (though I had 24, this ensures that writes and reads are larger than the cache, so caching has a negligible effect on the results). I ran each test three times and averaged the results.

There is a big surprise in this data, which I will have to revisit: the sequential block read performance for 6 disk RAID5. Is that an anomaly in my configuration, or is it really that fast? I will need to revisit that when I set the test environment up again. I would expect results more consistent with RAID6 read performance, but perhaps RAID6 isn’t as mature as the RAID5 algorithms.

This isn’t as complete as it could be, and other disk benchmarks, like iozone, do a better job of characterizing disk performance with random workloads, where the SSD would likely do much better. There are also newer disk controllers out there, namely the Dell H700 with 6 Gbps SAS links, that may improve on these scores. But it’s what I needed for something I’m doing, and if it helps someone else I’m glad I posted it.

Click on the graphic for a larger version that’s more readable.

Never Send Error Email in a Loop

Bob Plankers — Wed, 24 Mar 2010 15:19:31 +0000

Some of my favorite system outages are denial-of-service attacks brought on by coders who code as if nothing will ever go wrong. For instance, take the following section of pseudocode:

foreach $email (@giant_list_of_customer_email_addrs) {
    @customer_info = get_database_info_for_customer($email);
    if (!defined(@customer_info)) {
        send_error_email_to_admins($email);
    } else {
        send_customer_email(@customer_info);
        undef(@customer_info);
    }
}

When get_database_info_for_customer() fails (such as when the database is down for maintenance), someone will get an email for every failure. This is merely annoying when @giant_list_of_customer_email_addrs is 50 people, but when it’s 200,000 people it’s a big problem. First, you get hundreds of copies of sendmail running (or whatever the mailer function uses — with a lazy coder like this it’ll usually be something that isn’t efficient at all). Second, your local SMTP server gets overwhelmed, and its spam & antivirus scanners start melting down. Third, your mail spool fills, which causes other problems, like bounce messages that exacerbate the situation. And now, instead of having your database server down for maintenance and a scheduled script that should have run but couldn’t, you also have some of your other systems down as part of an incident. Whoops.

Never send error email in a loop. If you need to notify someone that a script had a failure do it outside of the loop, and use a stack to keep track of the errors that were encountered to be listed in the error message. Certainly, in this case, detecting a database problem and exiting would have helped, too. However, with all the things that could go wrong with databases, customer information, and general programmatic input it’s still worthwhile to avoid the inadvertent DoS by moving error notification outside the loop. At least that way you only risk having a single, potentially giant, email. :)

Future Capacity Planning

Bob Plankers — Thu, 21 Jan 2010 22:10:17 +0000

My favorite question from manager types is:

“How many more VMs can we run before we have to expand?”

I can never answer this without someone sticking it to me later. I always do end up answering it, and my answer is always wrong because it’s based on averages and the very little I’m told about future projects, upcoming P2Vs, server replacements, etc. We aren’t going to get 25 more 1.28 vCPU/2.398 GB of RAM VMs, though. It’s like having 1.75 kids — it just doesn’t work that way. I could try to tell them that we have 108 GB of RAM available, but that isn’t what they want, either. They want a concrete number they can multiply by our chargeback rates and put in the budget.

It’s hard to explain the problem with all of this, though, and I’ve been searching for a good analogy to make people realize why I’m so cagy about an answer. My awesome financial analyst, Michelle Fritze, just came up with it:

“How many boxes fit in your office?”

I can’t wait to ask my CIO that.

WebEx & Aero

Bob Plankers — Fri, 15 Jan 2010 20:02:38 +0000

WebEx and Microsoft Windows 7 don’t seem to get along 100% quite yet. If you are using WebEx on Windows 7 it’ll disable Aero during your session. However, if your session is over and you don’t get Aero back here’s how to fix it without rebooting:

1. Make sure you’ve closed/exited all WebEx components.

2. Right click on Windows Menu->All Programs -> Accessories -> Command Prompt and choose “Run as administrator.” You will need to accept a User Access Control warning about this.

3. Issue the commands:

net stop uxsms

net start uxsms

That should fix it.

Alternately (and potentially easier): you could restart the “Desktop Window Manager Session Manager” service via the Services administrative tool.

Playing Mastermind With My RAM

Bob Plankers — Tue, 12 Jan 2010 19:46:02 +0000

I have a Dell PowerEdge R610 in one of my VMware vSphere clusters that has been reporting memory errors. In fact, the machine wouldn’t boot, and the front panel suggested I reseat all the RAM. Okay…

0. Reseat all the RAM. Didn’t work, as expected.

1. Pull all twelve DIMMs out, put four back in. That worked, machine comes up.

2. Put four more DIMMs back. That worked, machine comes up.

3. Put last four DIMMs in. Machine doesn’t boot, same original error.

4. Pull last set of DIMMs out. Boot machine. Notice that BIOS is really old. Upgrade BIOS, thinking this is some stupid BIOS bug. Machine continues to boot.

5. Put last four DIMMs back in. New BIOS actually tells me what DIMMs are bad. Nice, except it says that A1 and A4 are bad. Two DIMMs? Yeah, not likely.

6. Order single replacement DIMM from Dell, decide to play Mastermind with RAM.

7. Replace DIMM A1. Machine switches to saying DIMMs B3 and B5 are bad. Really? DIMM banks B are on the other CPU.

8. Stifle disbelief, take loose DIMM from A1 and replace B3.

9. Machine switches to saying DIMM B5 is bad.

10. Take loose DIMM from B3 and replace B5. Machine likes that, has all of its RAM again, and I probably have the offending DIMM out now. Probably.

Lessons here: A) physical hardware sucks. B) linear troubleshooting rules. C) keep your firmware up to date.

The Lone Sysadmin » System Administration

Three Organizational Decisions That Help Me Virtualize

Related Posts

Happy System Administrator Appreciation Day!

Related Posts

Rate-Limiting Steps

Related Posts

Why No-Reply Email Is A Bad Idea

Related Posts

More Equipment Means More To Go Wrong

Related Posts

Levels of Indirection

Related Posts

Midnight is Always Tomorrow

Related Posts

What are P-states and how do I use them in vSphere?

Related Posts

If You Don’t Like Change…

Related Posts

Power Consumption of a Dell PowerEdge R610

Related Posts

Dell PowerEdge R610 & PERC/6i Disk Comparison

Related Posts

Never Send Error Email in a Loop

Related Posts

Future Capacity Planning

Related Posts

WebEx & Aero

Related Posts

Playing Mastermind With My RAM

Related Posts