Let’s just say that if you’re running VMware Virtual Infrastructure 3.5 Update 2 you probably can’t power your VMs on anymore. DOH. Unfortunately, that’s me. I updated everything on Sunday after testing for two weeks, and I can’t even imagine how I’d test for this.
The whole idea of patching sucks. There are always bugs, and you always trade one set of bugs for another when you upgrade. Of course, you use testing to try to figure out if there are more bugs or less, but things like this always show up. I’ve been meaning to write a longer post about patching, especially in the wake of this DNS debacle, but Michael Janke’s post “Patch Now – What Does It Mean?” over at Last In, First Out covers most of what I wanted to say. Especially about security researchers calling for immediate action:
When security researchers/bloggers announce to the world ‘patch now’, are they are implying that the world should ‘patch now without consideration for testing, QA, performance or availability’? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?
Amen. Overall a nice, thoughtful way to present it, and worth the couple minutes to read.
“Amen. Overall a nice, thoughtful way to present it…”
Gosh, I couldn’t disagree more. Any competent infosec expert who says to ‘patch now without consideration for testing, QA, performance or availability’ should have their badge revoked. Since security == confidentiality, integrity, and AVAILABILITY, it’s implied that proper change management procedures be followed. In fact, in my experience it is the infosec team that drives most change management – not the system/network folks.
Ben –
I was quite surprised at the number of self proclaimed ‘infosec experts’ with large blog followings, who did not consider availability when advising immediate patching on the recent DNS fiasco, even after it was demonstrated that the patch did work under high loads. Some of them made a big deal out of the fact that their upstream ISP didn’t patch fast enough for them, implying incompetence at the ISP rather than considering the possibility that the ISP was testing/QA’ing, or the possibility that the ISP tested and figured out that the patch didn’t scale.
Hence the post.
My guess that that you’d classify them as non-experts, which would be correct. They classify themselves as experts though.
But then – they probably aren’t reading either one of these blogs, so they still will not get the message. 😉
–Mike
The test for this, and you should be adding this into your QA or certification, is to set the date a year in advance, then 3 years, then 5 years.
This is a pretty standard QA technique. It scare the bajezus outta me that VMWare missed this.