The Lone Sysadmin » Virtualization

Free Upgrade to 25 GB for Microsoft SkyDrive Users

Bob Plankers — Sat, 28 Apr 2012 05:40:26 +0000

I don’t know all the details, so it may not apply to everybody, but if you’re a SkyDrive user (or just have an account) you might be eligible for a free upgrade from 7 GB to 25 GB (I haven’t heard of anybody not being eligible, though).

Log in to https://skydrive.live.com/
Click “Manage Storage” on the bottom of the left navigation column.
Click the magic button to upgrade your SkyDrive Free plan from 7 to 25 GB.

It is my understanding that this is a limited-time offer, so get on it. It takes about 20 seconds if you know your password. :)

No related posts.

Did you like this article? Please give me a +1 back at the source: Free Upgrade to 25 GB for Microsoft SkyDrive Users

This post was written by Bob Plankers for The Lone Sysadmin - Virtualization, System Administration, and Technology.. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2011. All rights reserved.

Thoughts on the VMware Code Leak

Bob Plankers — Wed, 25 Apr 2012 20:33:58 +0000

VMware has confirmed that there was a theft of the ESX source code around April 8th, 2012. I have some non-linear thoughts on this whole thing.

First, the code is from 2003 & 2004, and for ESX. ESX was the big, bloated hypervisor that shipped with whole Linux installs, and the purported 300 MB of source code sounds like they might have code for a lot of the Linux utilities that shipped with. So what? The newer version is ESXi which forgoes the Linux install in favor of being very small. That said, I’m going to assume they have the source code for the base hypervisor itself. I’m also going to assume that some of the hypervisor code from then has made it forward into current products, which would make everything vulnerable if there is a bug found.

I can only assume that VMware is doing internal code reviews and using automated security testing tools to find at least the basic problems in their software, like null pointer dereferencing, buffer overflows, SQL injections, etc. Certainly if they are not doing this now would be a good time to start to get ahead of this problem.

I’m also going to assume that if a problem is found that it won’t be made public by the hackers. There’s big money in the sorts of things you can do with a vulnerability in software like this. As such, it’s an arms race now, and whoever finds the problems first wins.

There are all sorts of problems a hacker could cause for an ESX host if a vulnerability is found, but many of them should also be prevented by your “defense in depth.” Defense in depth is the idea of using multiple security controls in layers so that when one fails, like in this case, another will be able to cover it. If a remotely exploitable vulnerability exists in ESX a properly configured firewall should be able to prevent exposure or compromise.

The holy grail, though, is a VM escape, where a VM can be made to interact with the hypervisor outside of the standard APIs. If you can find one of these you can compromise anything you have a VM running on, and potentially access your neighbor VMs, which is bad news for cloud providers and multi-tenant setups. These are very rare, though, and very high-profile when they’re found. Frankly, it’s much more productive & possible to attack infrastructure in other ways.

Beyond security, though, there are also industrial espionage implications, in that you get to see the way the virtualization industry leader is doing proprietary things like vMotion, etc. Hopefully their competitors are already well past 2003 & 2004 levels of code, but who knows.

The biggest and most persistent problem is all the enterprises that just don’t update their software. All OSes & software need updates, and your failure to acknowledge and prepare for that fact is not the vendor’s fault, despite an event like this. A proper configuration for vSphere clusters should allow vMotion, which then allows you to do rolling updates of the underlying infrastructure without affecting the workloads once a patch is made available. Which in turn means you should be doing these updates.

So, in short, this is a serious situation, but it’s no different than using an open source hypervisor where everybody can see the code. If you’ve been using good security & system design practices it isn’t something to freak out about, but I’d be prepared to apply some security-related releases in the near future. And if you aren’t there now I’d be thinking about getting to the latest levels of the major version you’re on pretty soon, because, if nothing else, it’s just a good habit to have.

Did you like this article? Please give me a +1 back at the source: Thoughts on the VMware Code Leak

My New Intro Slide For Cloud Presentations

Bob Plankers — Wed, 25 Apr 2012 16:00:26 +0000

I think I’m going to start using this slide in presentations on cloud computing. I’ll just cross out “Afghanistan Stability” and write “IT.”

I particularly like the “insurgents” and “narcotics” sections. Seems about right.

Did you like this article? Please give me a +1 back at the source: My New Intro Slide For Cloud Presentations

Upcoming Virtualization & Sysadmin Events in Wisconsin

Bob Plankers — Tue, 24 Apr 2012 04:36:43 +0000

There are a number of virtualization & sysadmin events coming up in Wisconsin in the next two months. Here are the ones that I know of, if you know of more please comment and I’ll add them.

Madison’s first ever #vBeers is Thursday, April 26, 2012 at the UW-Madison Memorial Union. I’m putting it together, if you’re in the area please stop by. It’s supposed to be in the upper 50° F range outside so we’ll probably be inside in the Rathskeller. No signup needed. Please come and keep me company.
Stephen Foskett is setting up Milwaukee’s first #vBeers for May 1, 2012, at the Miller Time Pub, 7 to 10 PM. You can sign up via Eventbrite. So far it’s Stephen, me, and Rod Gabriel of WI VMUG fame. Should be fun. There’s also a Chicago vBeers on April 30th if you’re closer to that.
Stephen and Truth in IT are doing a seminar on Building Virtual Infrastructure on May 2, 2012 in Milwaukee. Stephen is a great presenter, very knowledgable, and the day is in the style of the immensely popular “Backup Central Live!” series that W. Curtis Preston does. The nice thing is that the sponsored parts aren’t overbearing, they’re only 30 minutes each, only about 30% of the total, and all excellent smaller companies doing innovative work. It’s all day, free for end-users, and you can sign up via their form. There are other cities on the tour, too!
The Wisconsin VMUG meets on Tuesday, May 15, 2012 at Promega Corporation in Fitchburg, WI. My friend Chris Miller will be there from F5, talking about scaling View, and my friend Keith Norbie from Nexus Information Systems will be there talking about vCloud Director. Should be a lot of fun. Register at the VMUG web site.
Tech on Tap will be covering SharePoint for their June 9th gathering at the Stone Cellar Brew Pub in Appleton, WI. I was part of their first event on virtualization and databases and it’s nice to have a sponsored beer or four while you’re watching a talk. The problem, of course, is remembering that I live 100 miles away…
The “Yet Another Perl Conference::North America” conference will be in Madison on June 13 through the 15th. If you use Perl and are within 300 miles of Madison you might consider attending. It’s damn cheap, highly relevant, and the sessions are by leaders in the Perl community (or, put another way, the people that invented the stuff you’re coding in, on, and to). Plus, you can stay in the dorms! :)

Be there or be square. :) And let me know what else is going on in Wisconsin, particularly the northern parts.

Did you like this article? Please give me a +1 back at the source: Upcoming Virtualization & Sysadmin Events in Wisconsin

How To Increase Your "% Virtualized" Rates

Bob Plankers — Thu, 05 Apr 2012 19:50:37 +0000

The #VirtualizeDell tweet chat today got me thinking about what stops most virtualization implementations around 50-75%. These are just some thoughts on ways to kick things loose. @LethaW commented “[that some of them are] sneaky and underhanded, and I love it.” I took that as encouragement. Needless to say, your mileage may vary.

Problem: Physical hardware is required or requested by vendors.
Solutions:

Actually check to make sure that a vendor does require physical hardware. For example, Oracle doesn’t require it for many things, but there’s this misconception out there that they do, and I hear it from DBAs a lot. Consultants will also tell you a wide variety of things, too. Check the facts. Get it in writing.
Don’t do business with vendors that require physical hardware. Get your purchasing people to adjust their RFP process so that virtual infrastructure support is a requirement.
Agree, then give them virtual servers anyhow and don’t tell them. Uninstall the VMware Tools if they notice (but put them back on later).*

Problem: App people want physical hardware because “it’s faster” or “it’s just easier.”
Solutions:

Point out that it’ll be three months before they can have physical hardware (considering the ordering process, shipping, racking, installation, OS configuration, etc.) or three days for a VM.
Make your chargeback scheme reflective of the additional costs associated with a physical server. If you’re doing it right a physical box should cost the requestor a lot more to operate than a VM.
Test. Put their stuff in a virtual machine first, then if they can show that they aren’t meeting stated & quantifiable (with numbers) performance goals, and there’s nothing that can be done in the virtual environment to improve performance, then they can have physical hardware. Usually they don’t have quantifiable performance goals so you win.
Agree, then give them virtual servers anyhow and don’t tell them. Refer to it as a “server” without the physical/virtual label.*

Problem: Physical hardware is “needed” because a vendor’s licensing is way more expensive for a virtual implementation.
Solutions:

Call the vendor and educate them on how your brand of virtual machines work. I usually have to tell them that a VMware VM doesn’t have access to more than the configured number of vCPUs, so a 2 CPU license on a 2 vCPU VM would be the same as a 2 CPU physical host. Blow their minds by telling them that you can change the number of CPUs in a physical server, too, either by disabling cores in the BIOS or adding additional socketed CPUs after the fact, so there’s no difference between a VM and a physical host in that regard. Use the term “hard partition.”
Don’t do business with technology vendors who know nothing about technology.
Don’t tell them it’s a virtual machine. Refer to it as a “server” without the physical/virtual label. Ask for a 1/2/4 CPU license for a physical machine, remind them that you can disable cores if they tell you there’s no such thing as a 1 CPU server anymore, and don’t tell them otherwise.*

Problem: Physical hardware is needed because someone wants to use a custom piece of hardware, like a line card or a licensing dongle.
Solutions:

Use DirectPath to assign the hardware directly to a VM.
Use a networked USB hub for dongles, cameras, serial-to-USB adapters, etc.
See if there’s another alternative that doesn’t require custom hardware. For example, do you really need a line card for your VoIP implementation, or can you just get a SIP trunk from someone?
A dongle? Really? 1991 called and wants its licensing scheme back so it can party with MAC-based licenses in FlexLM’s basement.
Custom hardware & DirectPath breaks things like vMotion/HA/etc. Consider just giving them a small physical host so they don’t mess up your whole support & infrastructure patching model. Change your % virtualized numbers to reflect “virtualize-able” apps, so you still have a 100% virtualization rate.*

Feel free to add more stuff in the comments. I always like conversations like this. :)

* Underhanded & sneaky. I take no responsibility for what happens if you use these ideas, but I suggest them wholeheartedly if the right way isn’t working.

Did you like this article? Please give me a +1 back at the source: How To Increase Your "% Virtualized" Rates

Technical Debt

Bob Plankers — Wed, 04 Apr 2012 20:04:29 +0000

This week I’m paying off technical debt. If you’re not familiar with the term it’s from the world of software developers, and Martin Fowler describes it better than I would:

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

System administrators and operations folks know this phenomenon very well, since we’re often called to make “temporary” fixes to things. As we all know, nothing is ever temporary, because if it works we move on. And it isn’t like we stopped to consider a proper design for a temporary fix, so we end up with something that helps us greatly in the short term but really stinks in the long term. Steve McConnell, whose “Code Complete” book was the first to introduce this term to me, has a great post where he also outlines the problems with technical debt:

One of the important implications of technical debt is that it must be serviced, i.e., once you incur a debt there will be interest charges. If the debt grows large enough, eventually the company will spend more on servicing its debt than it invests in increasing the value of its other assets.

Soon you find yourself unable to move forward because you’re spending all your time servicing your debt. Sometimes it gets so bad that patching & updating stops, changes are frozen, all because something might tear loose the duct tape, gum, and string holding the infrastructure together. As Jeff Atwood puts it, “accumulated technical debt becomes a major disincentive to work on a project.” He’s totally right. As a result, over time, we end up with systems that are seriously decrepit because nobody wants to touch them. Eventually something comes along to knock the whole thing over.

That’s what I’m doing this week: knocking things over on my own terms. I’ve been keeping a list of all the things I hate about my environment. It’s mainly a series of little things, with just a few big things. I’m writing new monitoring scripts for my VMware environment so that we don’t get paged when we reboot a host that was in maintenance mode, and that we do get paged when DRS isn’t in fully automated mode for more than a certain time. I’m de-kludging the VMware Tools installs on Linux because VMware fixed most of the problems we’ve had with them. I’m going through all the hosts and fixing the firmware levels, because they’re a mess, and converting the last of my oddball hosts to our standard configurations.

This work is at the short-term expense of other projects, but it’ll be worth it because I won’t have to dedicate time, energy, brain cells, and documentation to all the exceptions. I’ll be able to focus on moving forward, not just running frantically to catch up. And it’s very easy to explain what I’m doing. As Steve McConnell puts it, the metaphor has an “incredibly rich ability to explain a critical technical concept to non-technical project stakeholders.” Turns out that most managers understand debt.

What technical debt do you have in your environment? Do you keep a list of things that would be nice if they were fixed? Why not? How much happier would you and your coworkers be if you could block off a few days of time to fix those things?

Did you like this article? Please give me a +1 back at the source: Technical Debt

Wisconsin VMUG in Green Bay – April 5, 2012

Bob Plankers — Mon, 02 Apr 2012 16:23:12 +0000

Just a reminder that there’s a Wisconsin VMUG gathering in Green Bay, WI this coming Thursday, April 5, 2012. Rod Gabriel (@ThatFridgeGuy) says that 80 people are registered so far, so the turnout will be excellent.

Scott Herold of VKernel/Quest will be there to talk about performance & capacity planning (ostensibly with the VKernel tools). Mike Chudzik from Pure Storage will be there to talk about smashing performance bottlenecks, ostensibly by using their storage devices which are all-flash storage arrays and pretty sweet. I’ve seen both of these vendors via Tech Field Day, and they have some interesting things going on. Last in my list (may not be last in the presentation order), Jeremy Gruenke of Johnsonville Sausage will be giving a presentation as well, which seems perfect for the sausage making that is IT most of the time. It’s too bad we can’t treat annoying applications like bratwurst: boil them in beer & onions and then stick them on the grill.

You can register on www.vmug.com for the event.

Did you like this article? Please give me a +1 back at the source: Wisconsin VMUG in Green Bay – April 5, 2012

Madison, WI vBeers – April 26th, 2012

Bob Plankers — Tue, 27 Mar 2012 18:51:09 +0000

I greatly enjoyed the three or four Twin Cities vBeers I attended last summer, and one of the things on my list for 2012 was to start a regular vBeers in my hometown of Madison, WI.

So I’m going to.

If you are in the area on April 26th, 2012 please stop by and join us on the UW-Madison Memorial Union Terrace (Rathskeller if it’s raining/snowing), sometime after 4 PM. If you know someone who can do something about the weather, we’d like some sun & warmth!

As I said in the invite (linked above) RSVP isn’t necessary, though if I know you’re coming we’ll save you a seat. Or flee the area… :)

Did you like this article? Please give me a +1 back at the source: Madison, WI vBeers – April 26th, 2012

vSphere 5 Update 1 and vCenter Agent Upgrades

Bob Plankers — Tue, 27 Mar 2012 15:51:04 +0000

We applied vCenter 5 Update 1 on Sunday in my environment, which is earlier than we often do because there’s some bugs we needed fixed (namely the issues with Fault Domain Manager/HA and SSL certificate replacement). We’ve been running Update 1 in our test environment since its release, and it looked solid. But as any experienced IT person will tell you, production isn’t test. Ever.

After the upgrade none of the hosts were connected to vCenter, and there was a task for each cluster entitled “Upgrade vCenter agents on cluster hosts” that did not proceed, did not terminate, and was uncancellable. The hosts themselves had a warning message on the summary tab indicating that a manual upgrade of the agent was necessary. I could manually reconnect the hosts, and it would do its thing with the agents, but if vCenter was restarted the tasks would reappear, with new start times, and everything would be disconnected again.

When I upgraded vSphere from 4.1 U2 to 5.0 I was given a choice to manually upgrade the agents, or do it automatically. At the time the manual upgrade seemed a better choice, since I was having upgrade issues. That choice seems to have flipped the vCenter Server Setting AgentUpgrade.autoUpgradeAgents to false:

Setting that back to true, reconnecting the hosts, and restarting vCenter fixed the problem.

I can’t take credit for this fix myself, as it was my VMware BCS guy Jonathan Proskey that thought of it, working off notes from an already open case/bug filed with VMware Engineering. But it’ll be a while until a KB article appears, and it’s extremely new, so with his permission I thought I’d post it for anybody who encounters the same issue. At least this way Google will return something for the phrase “Upgrade vCenter agents on cluster hosts.” :) If anybody needs it, it’s SR 12157263203.

Did you like this article? Please give me a +1 back at the source: vSphere 5 Update 1 and vCenter Agent Upgrades

More Brains, Fewer Crotches

Bob Plankers — Sat, 24 Mar 2012 19:11:26 +0000

Once again we’re back on the sexism topic in tech, watching people with big social media spotlights shine light on terrible ideas and stupid moves. This latest round has been a double-whammy: Squoot treating “women as party favors” (to borrow great wording from the title of Rachel Balik’s Forbes article, linked below) and Geeklist reacting poorly to an inquiry about an advertisement ostensibly done in their name. Add these to the stack of other companies consistently hiring spandex-body-suit-clad booth bunnies at conferences, GoDaddy, and the numbers at Gina Trapani’s “Narrow the Gapp” and it’s obvious that there’s a problem on many levels.

Why I’m really here, though, is because I got asked via Twitter DM “what does this have anything to do with you? tired of people captializing (sic) on the situation!!”

I don’t make business decisions with my crotch, that’s what, and I’d appreciate not being treated as if I do.

Yes, I am capitalizing on the situation. Strike while the iron is hot. Someone in a position to do something should do something, too. For me, right now, that “something” is pointing out that the business tactics that rely on sex just aren’t a good business decision anymore. Not only are these approaches insulting to women but they’re insulting to men as well, and increasingly likely to get called out in very public ways. Why? Because being a person in technology, male or female, is largely about brains. Just like women, us guys also don’t like having our brains ignored while you focus on our crotch. We’re not asexual by any means, but we see what you’re doing and we are smart enough know that there’s no way we’re going back to our hotel room with a booth bunny. Thing is, now we’re not taking your product demo back there, either.

For every company I can think of doing the wrong thing there’s one appearing to do it right, though, and I don’t like seeing everybody lumped in together. I love walking up to the Compellent or Symantec or HP booths because the “booth babes” are actual employees who know what the heck they’re talking about, and often are the people designing & building & supporting what I’m interested in. Using their enormous brains, answering my questions, having a conversation, showing off their work, creativity, and ideas — that’s hot. These are the women I want to go have a drink with!

As fun as it can be to get into other people’s pants, it’s much more fulfilling to get into their heads. Guy or girl. So, as an industry, let’s have a little less of the body and a lot more of the mind, eh? As The Oatmeal dubs it, a “brobdingnagian brainboner” lasts longer and is probably the conversation you really wanted to have, anyhow.

Here are the links to things I’ve referenced, and I welcome civil & thoughtful comments below. I know I’ve made some generalizations and no two people are alike. This is my opinion about the situation, from my point of view, and others have theirs, which aren’t wrong, either. I tried to stay out of the whole Geeklist thing, because as I told my friend @gallifreyan on Twitter the more I look at the situation the more I see that everybody lost in some way (though I do applaud Basho for their response, which was mainly that it has nothing to do with them). GeekList wasn’t specifically my point here and I think Scott Hanselman properly sums up many of the lessons I took from the event.

Rachel Balik: Women As Party Favors And Eye Candy: The Latest Tech Marketing Meme
Gina Trapani: Narrow the Gapp
Charles Arthur via Storify: OH HAI SEXISM
Scott Hanselman: Why You Should Never Argue in 140 Characters or Less
Matt Simmons: Seriously, stop with the booth babes
Stephen Foskett: (Don’t) Make Your Startup Look Stupid With Booth Babes & Chotchkies!
I realized I am probably stealing the “get into their heads” line from Dan Le Sac Vs. Scroobius Pip’s “Thou Shalt Always Kill.” Love that song, and now it’s not plagiarism.
The Oatmeal is awesome. “Brobdingnagian brainboner” is from “Being in love at age 27.”

Did you like this article? Please give me a +1 back at the source: More Brains, Fewer Crotches

Rackspace’s Terrible Maintenance Plan

Bob Plankers — Wed, 21 Mar 2012 01:13:06 +0000

Update, 3/21/12: please read the comments, too — we got a good response from one of Rackspace’s folks.

I got a note today from Rackspace, where I have two virtual servers in their Rackspace Cloud. It was opened in the form of a support ticket, waiting for input from me, but with the text of the support ticket labeled as if I entered it, which was weird.

As part of our ongoing effort to provide you with the best Cloud Servers service possible, we routinely perform maintenance and upgrades of our underlying systems. The majority of these are performed non-disruptively, however maintenances sometimes arise that impact Cloud Servers instances. At this time, a Cloud Servers host update is required that will involve an automated migration (i.e. relocation to a new physical host server) of some cloud servers including the following server(s) in your account:

This makes sense. Over time, hardware gets old and should be replaced. Amazon just did this, too, forcing a lot of people to reboot their stuff, and in the restart process it’ll find its way to hardware & hypervisors from this decade. Shouldn’t be a big deal, I do this at work with my virtualized stuff, too, and most people can work it into their normal maintenance.

Preferred Option: MANAGED MIGRATIONS – Rackspace managed and controlled

After March 21st, 2012 at 11:59PM CDT, managed migrations will begin being scheduled for any cloud servers listed above. If you have multiple servers listed, it is likely that they will be spread out across several days beyond March 21st. You will be notified 24 hours in advance of any managed migrations and will be provided a specific time window in which the migration will occur. Managed migrations require no effort on your part, will automatically be performed by Rackspace, and will effectively appear to you as a reboot…

Alternate Option: AUTOMATED SELF-MIGRATIONS – One click migrations you control

To allow you to migrate at your convenience and minimize impact to your applications, you may perform your own automated self-migrations anytime between now when your servers are scheduled for a managed migration. This process is simple, can be performed with the click of a button, and will effectively appear to you as a reboot.

The plan seems sound, but today is March 20, 2012. From the time this made it to my inbox I had only 57 hours (33 + the 24 they give you prior to doing it for you) to reboot my stuff on my own terms? Are you serious?

While I’m reading this whole thing an update comes in:

We apologize for the miscommunication. The date on the initial update is incorrect. Managed migrations will not begin on these servers until after March 27th, 2012 at 11:59 PM CST.

Now I get a week to take my stuff down on my own terms, which, frankly, isn’t a whole lot better. What if I don’t have a maintenance window in the next week, or staff availability, or any other good reason why I can’t or shouldn’t do the work in that big of a rush? This becomes an unplanned emergency for me now, for no good reason as far as I can tell.

Here’s my take on this whole situation:

A day’s notice is irresponsible & asinine, a week’s notice is ridiculous. A month is better. Two months would be nice. Give people ample time & notice to take care of the situation themselves, then force the stragglers into compliance. There’s a darn good chance that in the next two months they’ll have a maintenance window anyhow. And it isn’t like the folks there at Rackspace haven’t known this was going to need to happen for a long time (or, if this was somehow a surprise, they need a new project manager). Send the stragglers a note every week for two months, and in the last email to them assign a firm date & time when they’re going to see a forced reboot.
Fix the CST/CDT issues in the notices so they’re accurate. Better yet, express time the way everybody else with a multi-timezone audience does: in 24-hour GMT. If you’re worried about people asking questions put a translation table into the FAQ.
While I think the support ticket idea was a good one, don’t open support tickets in my name with initial text credited to me that I didn’t enter. At the best it’s confusing, and I’m the kind of guy that doesn’t like being credited with things I didn’t do. The followup correction was from “Support at Rackspace Cloud…” so it’s obvious it can be done. Choose to do it right.
Get someone who is detail-oriented to read the notices you send to your customers prior to sending them, to vet the whole plan, and perhaps play the devil’s advocate. I suspect there’s someone in the Rackspace Cloud support organization that could have provided all this same feedback, internally, if they’d had the chance, but now you have an annoyed customer doing it for you. Nice shot.
Do a better job of highlighting that this process won’t be instantaneous, that it requires a soft reboot from the Rackspace Cloud control panel and not from within the OS, and that people should go read the FAQ for more details on how long it’ll take. The notice could have easily been more informational, though the FAQ did do a good job of indicating to me what was going to happen.

I’m comfortable saying that if this were a change request I’d filed in my place of employment it would have been denied by our change managers based on lack of timely customer communication for a non-emergency change and inaccurate details. C’mon Rackspace, you can do much better, and you need to if you want enterprises to move any workloads in your direction after this.

Did you like this article? Please give me a +1 back at the source: Rackspace’s Terrible Maintenance Plan

Three Failings of Virtual Appliances

Bob Plankers — Mon, 12 Mar 2012 18:56:37 +0000

I’m torn when it comes to virtual appliances. I love them because they make a lot of installations absolutely brainless. I hate them because the people that create them make assumptions that are ridiculous and unsupportable. Here’s the three ways I hate them the most:

1. There’s no good way to back them up.

For organizations that haven’t gone the “whole VM” backup route there are very limited choices for backing these things up. Sometimes the virtual appliance has some method to export the configurations and data, but often not. And when there is a method it’s usually a web interface that cannot be automated.

What I want: virtual appliances should be able to export their configuration and data on a automated, scheduled basis, to a Windows share and via SCP. Since nearly all virtual appliances are Linux-based this shouldn’t be a big deal, since Samba is readily available, and SCP is a staple of OpenSSH. Both options would be nice but if you have to pick one pick the Windows share, since even Linux folks can run Samba, and it’s probably easier to script on the appliance side (you can write directly to it, unlike SCP where you would have to have a temporary holding spot). Scheduling isn’t an issue, either, since Linux has cron & at.

2. They want to DHCP.

My data center doesn’t use DHCP. Many places don’t. A VM that assumes it will be able to DHCP is a giant pain. Some do it right and allow you to configure a static IP after the boot process has completed, via the command line or console. Some do it poorly and only permit you to change the IP from a web interface. How exactly do I get to the web interface if the VM doesn’t have an IP? I end up having to run a DHCP server on an internal dvSwitch portgroup just to configure the VM.

What I want: virtual appliances should always allow people to easily configure static networking addresses from the console, for both IPv4 and IPv6. And the IPv6 stack should not be enabled unless it’s specifically configured by a user, either for autoconfiguration or static configurations. It also needs to allow full control of the classic settings, like gateway, address, etc.

3. They are seething pits of evil, security-wise.

Let’s just face it, virtual appliances never get patched. They’re giant security holes, and I’ve had a few of them get hacked when they’ve been on a public-facing network, using vulnerabilities that were patched six years prior. You can’t firewall them to protect them, either, because the holes are in the applications (which you have to grant access to). Updates are never released in a timely fashion, and there’s usually no working way to update the virtual appliance. So, in my opinion, as of this writing, you cannot use any virtual appliances in any public-facing manner.

What I want: virtual appliances should be updated no less than quarterly so they contain the latest patches. People who create virtual appliances should use the latest patch levels, and scan their products prior to release with Nessus in order to check for vulnerabilities. It would be nice if you allowed end users to update the appliance, after allowing them to export the configurations & data (see complaint #1 above). Full host-based firewalls should be enabled on all virtual appliances, for IPv4 and IPv6, for ingress and egress traffic, and nothing should be permitted in or out unless it’s absolutely necessary. Last, an appliance should have no software installed on it that isn’t 100% necessary for the function of the appliance, or support. Not wasting my disk space is just polite, plus the less software you install the less of a security profile I have to worry about.

So, in short, if you’re a virtual appliance creator I wish you’d get it together! I’d love to see many products shipped as appliances, but I cannot run them until you fix some of these operational issues.

Did you like this article? Please give me a +1 back at the source: Three Failings of Virtual Appliances

Dell PowerEdge 12G Is Here

Bob Plankers — Tue, 06 Mar 2012 19:08:37 +0000

Over the last week there’s been a number of different posts about the new Dell PowerEdge models, the 12th generation (12G) of their server line. I was briefed both by Dell technical staff and by Dell executive staff on the Rx20 lineup and I took a few notes. I was mainly briefed about the Dell PowerEdge R620, R720, R720xd, which will be in the first wave of refreshes. The higher-end models, like the R820 and R920, and the cloud & HPC focused C-series, will be part of another release soon after, and reach into the higher-end E7 CPU models (8 way, 10 cores) from Intel.

The new mid-range hosts are built around the Intel Xeon E5 CPUs, also known as “Sandy Bridge.” These CPUs are 4, 6, or 8 cores, and the the first number in the CPU model number is the number of sockets they will support. An Intel Xeon E5-4600 is a four-way CPU, E5-2600 is a two-way CPU, and an E5-1600 is a one-way (uniprocessor) CPU. Anecdotally, these CPUs are reported to turn in a 70% general workload performance improvement over the last generation of CPUs, and a look through spec.org at the benchmark numbers reported on E3/E7 hosts reinforces that.

Memory is quad-channel now, up from triple-channel in the Intel Xeon 5600 (“Nehalem”) line. This lets them run RAM at up to 1600 MHz, and it falls back like the Nehalems did to 1333, 1066 and 800 MHz. The speed depends on the memory configuration & type of DIMMs, like before. There are now 16 total DIMM sockets, up from 12, due to the extra channel. Peter Bailey of Dell points out that you can now feasibly (i.e. not 800 MHz) & economically (16 GB DIMMs only have a 10-15% premium now) put two 16 GB DIMMs on each channel of each CPU in a dual-socket R720, causing a sweet spot of 256 GB of RAM per host.

The R820, and presumably the R920, will have similar architectures, but increase the number of DIMM sockets to 48 from 32. As a current R810 owner I have no idea where they plan to put all those sockets, but it’ll be awesome to be able to get 1.5 TB of RAM in a 2U host. Perhaps they are using technology from the TARDIS, or they have Hermione Granger casting undetectable extension charms on the systems[0]. Regardless, the R620 also magically gains a PCIe expansion slot, bringing it to a total of 3. For many that means that an R620 is an effective replacement for an R710, saving 1U of space but giving us as many card slots.

The 12G hosts are all PCI Express 3.0, which means that bandwidth and the number of PCIe lanes has gone up. PCIe 3.0 bumps the number of gigatransfers per second to 8, from 2.0′s 5 GT/s. It also changes the encoding from the 2.0 spec’s 8b/10b encoding to 128b/130b, which reduces overhead. Between the two improvements, PCIe 3.0 “effectively delivers double PCIe 2.0 bandwidth” according to the Wikipedia article. From a practical point of view that means that PCIe 3.0 devices can all move down a slot size, so if you needed a x8 slot for your dual 10 Gbps PCIe 2.0 NIC the 3.0 version of that NIC will be x4. Good news for GPUs!

On the storage front there’s a few new improvements. First, the RAID controllers have all been refreshed. Marketing didn’t get the memo, though; the 11G models were the H700/H800, the 12G models are H710/H810. Perhaps with the 13th generation they’ll stop working to confuse everybody and name them the H730/H830… The controllers continue to support CacheCade, the LSI software feature that lets you use an onboard SSD as cache to front-end slower traditional disks attached locally. It’s essentially a direct-attach storage version of the NetApp FastCache cards, and a feature that’s finding its way into most disk controllers, including those built into Intel desktop disk controllers, too. I’d expect that the CacheCade restriction of only working with Dell-supplied SSDs will continue to be enforced, if only for reliability & support reasons.

Drive carriers are the same between the 11G and 12G lines, which is good because the Dell PowerEdge R620 will be able to hold 10 2.5″ drives. The R720 can hold 16, and the R720xd, whose extra letters stand for “extra drives,” can hold up to 24 in front and 2 in back. Sure, you don’t have a front LCD panel or DVD-ROM anymore, but nobody cares in 2012. I suspect that this will be increasingly useful in the face of technologies like the VMware Virtual Storage Appliance (VSA), where you could potentially have a two or three host cluster that has no external storage whatsoever, especially when coupled with the 10 Gbps networking & CacheCade options. At the very least I can recycle a bunch of the orphaned 146 GB 10K disks I’ve abandoned with my vSphere 5 switch to USB sticks for booting.

Beyond traditional SSD, though, is the idea of the PCIe-based flash card, like what Fusion I/O sells. You’ve been able to order Fusion I/O cards with Dell servers for a while now, and they’re incredibly fast because they don’t have all the overhead of RAID. However, those cards suffer some serious drawbacks in that they’re internal and not as serviceable as a traditional drive. In response, Dell has a build-time option that puts two or four PCIe hot-plug slots in the front of the chassis, so that you can have flash modules installed on a host that are more easily serviceable. This is a build-time option that consumes space on the front of the host, so if you think you want to add one of these modules later you need to order the server with the right options initially. You can’t retrofit it later.

The main complaint about SSDs is always the limited number of write cycles before failure. Dell is now saying that the SSDs they ship will withstand 12 PB of writes over their life. Figuring a 5 year lifespan of a server, that’s 6.73 TB of writes a day per device. Though I don’t know what the PCIe flash pricing is yet, prices on SAS SSD drives have fallen 30% or more over the last year, making all of this more affordable against traditional drives.

One big use for local SSD is local I/O caching, especially as a read cache for enterprise disk arrays. EMC has been talking about this with some of their recent releases, and other vendors, like Dell, are saying that they’ll release similar offerings later this year. Local SSD caching is a newer feature even in VMware products based on vSphere 5, and if the trend continues it promises to seriously curtail the amount of read I/O our shared storage arrays have to do. That ought to make storage administrators very, very happy, especially since virtual environments are all random I/O. It’ll be interesting to see the economics of this. Like most new features that stand to cannibalize enterprise sales I expect this to be an expensive add-on, and possibly a Dell Storage-only feature, but we’ll have to wait until the release to know for sure. I’m hoping it’s a more generic caching solution that’s reasonably priced, offsetting my organization’s spending on SAN and array improvements.

The 12G servers have all new network options as well. There is no longer the idea of an “onboard” set of NICs, as everything has been moved to a daughtercard. The upside to that is customers now have choices, Broadcom or Intel, all copper or some SFP+, and FCoE support like that of the X520 cards. This makes me very happy, because I’m an Intel NIC fan, and I’m moving in an all-10 Gbps direction, though this does mean my beloved FCoTR is falling behind again.

The minimum OSes needed to install on these servers are Microsoft Windows Server 2008, Red Hat Enterprise Linux 5.7, and Red Hat Enterprise Linux 6.1, though 6.2 is strongly recommended. Newer releases of Fedora, Ubuntu, and all the RHEL clones will work, too. This is merely due to driver support for the Sandy Bridge hardware. Fans of this blog and people who have attended any of my presentations know that I am a big proponent of keeping your OS somewhat recent. This is a big reason why. Dell merges their driver support with the mainline Linux kernel, so as long as you can run a newer kernel you should be in good shape. Add that to their yum repositories for OpenManage and BIOS updates and excellent history of support for Linux and they continue to have a great platform for open source computing. In the briefings I had, Dell’s Linux support was described as “boring” only because it just works, without fanfare or hassle. I like things that just work, perhaps because it’s so rare in IT.

On the management front Dell is releasing OpenManage Essentials, which is a streamlined hardware management and monitoring solution that is free to customers. Classically, OpenManage has been a real bear to install and maintain, and hopefully this makes it more useful as a tool. The 12G servers can be monitored & updated completely out-of-band. This is a giant win, as agents make everybody’s lives miserable. Of course, you still need an agent for all your older Dells, but it’s a promise of good things to come.

Similarly, they’re releasing the free & standalone OpenManage Power Center, where you can manage and monitor power consumption on all the 11G and 12G hardware you have. Details are a little scarce, but this is probably similar to HP’s Insight Power Manager, where you can administer power caps and monitor usage.

Dell also continues to support & expand their Fresh Air initiative, which certifies certain models and configurations of Dell hardware to work continuously at temperatures up to 95° F (35° C), 900 hours a year at 104° F (40° C), and 90 hours a year at 113° F (45° C). This has serious effects on data center design & operations, because they estimate it saves $100K per megawatt of OpEx and $3000K ($3M) per megawatt of CapEx for new data centers. For us it means we can turn up the temperature a bit as we retire all the older server hardware. It also means that some types of HVAC maintenance can now be done with the systems online.

That’s all I had for notes. If I’m missing something, or I’ve got an error here, please leave a comment. I owe a big thanks to Dell’s Peter Bailey, who gives an excellent, high-bandwidth, pragmatic, non-marketing presentation on all of this, and while he steals some of our jokes I’m stealing some of his numbers. I also owe a continual thanks to Andrew Waxenberg and Pat Meyers, my local Dell guys, a solid team. Likewise, I thank David Gibbs from Blanc & Otus for setting me up with the Dell Global Channel executives to get some of my 12G questions answered. Kevin Houston, a Dell engineer, has great content about the Intel CPUs and M-series of blades at bladesmadesimple.com. I first met him at a Gestalt IT Tech Field Day, which Dell has sponsored, and I’ve attended to great enjoyment and learning, in the spirit of disclosure. Dell has also given me access to seed & evaluation units in the past as part of doing business with them. This doesn’t generally skew my opinion, as most people who know me can attest to. Good technology gets a good review from me, bad ideas get skewered indiscriminately. :)

————

[0] Or a Bag of Holding, for AD&D fans.

Did you like this article? Please give me a +1 back at the source: Dell PowerEdge 12G Is Here

Is it possible to have too much monitoring?

Bob Plankers — Mon, 05 Mar 2012 17:06:29 +0000

In the category of shameless self-promotion[0] I’m one of the SolarWinds thwack Ambassadors for the month of March, and I’ve posted my first discussion topic, asking whether it’s possible to ever have too much monitoring.

If you’re not familiar with thwack it’s the SolarWinds community site, it’s great. I’ve been in orbit around SolarWinds since Networking Tech Field Day #1, and I’m happy to take a few topics that are more discussion-oriented and post them over there. SolarWinds also has a whole bunch of free tools there, like a Wake-On-LAN tool I’ve used to wake my VMware ESXi hosts up from standby, a VM-to-Cloud calculator to let you see how expensive it would be to take all your VMs and run them in the cloud, and a subnet calculator I recommend to newer admins to check their work. No problem seems wackier than a miscomputed netmask, after all.

So please, go check it out and tell me what you think (spoiler: I really think that yes, it is very possible to do too much monitoring).

[0] I really don’t do much self-promotion, but hey, if you like what I write here you might be interested in what I’m doing over there, too.

Did you like this article? Please give me a +1 back at the source: Is it possible to have too much monitoring?

Take A Break

Bob Plankers — Tue, 28 Feb 2012 18:38:02 +0000

I recently returned from a two week trip to New Orleans, for rest & relaxation. And it reconfirmed for me my suspicions that sometimes the best thing I can do, from a process & procedure standpoint, is to leave for a while.

Banks usually have a mandatory absence policy as part of their internal controls. The United States’ Federal Deposit Insurance Corporation (FDIC) encourages all financial institutions to require employees to take no less than two weeks of vacation every year:

During this time, their duties and responsibilities should be assumed by other employees. This basic control has proven to be an effective internal safeguard in preventing fraud. In addition, such a policy is viewed as a benefit to the well-being of the employees and can be a valuable aid to the institution’s overall training program.

I think this is solid advice, not just for financial institutions, but for IT as well. When people on my team go on vacation it exposes gaps in process, causes documentation to be updated, and just generally cleans a lot of little things up. In my case, I returned to a concise list of things that needed my attention, what one of my coworkers dubbed my “honey-do” list. Ha!

The trick for IT guys is twofold. First, to pull this off your duties and responsibilities need to be completely assumed by others. No checking your email from the hotel. No doing work from a cafe. I usually tell my coworkers to call me if they get really stuck and it can’t wait, but otherwise I let them do my job in my absence. No pagers, no email. If you’re gone you’re gone.

Second, your coworkers cannot treat any problems they encounter as some giant failure on your part. And similarly, you should not take a list of problems they found as a personal attack on you. Going on vacation should be restful for you, and educational for everybody, and when you return you can use that education to improve all the processes and procedures. No blame, no finger pointing, no apologies. Just agree that there’s some stuff to fix, then fix the problems and move on.

I can just hear some of you asking “But what if I’m the only IT guy?” Sure, smaller companies with a single IT guy have different situations. Even then, there are ways to take a no laptops, no pagers vacation. Is there another employee that is good with this stuff, that you could document most routine operations for? What about a local consulting company that could step in for two weeks if needed? Don’t feel guilty about proposing this, or setting it up. It’s good for your company. After all, what would happen to your company if you were hospitalized for two weeks and they had no choice in the matter? Having a plan for you to be on vacation is just a way to responsibly plan & practice for more serious situations.

Downtime is something everybody needs in order to be a better person. It’s good for the business because it makes them more resilient overall, and certainly more prepared if something were to happen to you. It’s cross-training for your coworkers, and makes you happy and relaxed. And besides, with something like 85% of outages caused by human error, how many problems will they actually have if you’re not around to mess with stuff? :)

Did you like this article? Please give me a +1 back at the source: Take A Break