The Lone Sysadmin

Stephen Foskett has a great post today on The Enterprise IT Acquisition Game, wherein he talks about how it’s open season on data storage companies, with a lesser emphasis on networking:

So this is the game: Four full-line enterprise superpowers battling each other for datacenter dominance and coveting the extra profits of a few verticals. HP clearly believes they can chip away at EMC and Cisco in storage and networking; Dell and IBM have so far focused mainly on storage; and Oracle hasn’t made a move in either direction, instead challenging the other three in the core server and software space.

Right on, especially with the “coveting the extra profits” part. For years, Dell, IBM, and HP have been busy commoditizing the compute node side of things. They’ve been driving the prices down on CPU and RAM for so long that the margins aren’t there anymore. And now, with the onset of widespread virtualization, the volume is no longer there to make up for it. However, storage is still a high-margin endeavor, and probably the single most expensive thing in my data center; I have $400K in CPU & RAM connected to $2m in storage. It is not surprising that there is a huge bidding war for 3PAR, and an open season on data storage companies. It’s one of the best ways for big companies to continue bleeding IT budgets, at least until SSD capacities rise and prices fall, and it does to storage what virtualization is doing to servers.

The thing is, virtualization is all about driving costs down in IT, and I just don’t see these big consolidation efforts as an actual way to do that in the long term. EMC has been downright nasty with my organization because they thought we were locked in. Cisco has done the same, and Oracle is generally regarded a bunch of jerks. Why would anybody choose UCS, VBlocks, or an all-in-one option from Oracle knowing that you’re locking yourself in? Even if the pricing is competitive right now, it won’t be for the fourth or fifth year of support where they think they’ve got you trapped.

Stephen’s last paragraph speaks of rebel alliances:

Although I would love to see a rebel alliance rise (imagine Juniper, NetApp, and Symantec joining forces!) this is not a likely scenario.

Like Stephen, I would enjoy seeing a rebel alliance, but I doubt it is likely, for two reasons. One, companies like buying from a single source, because they can get support guarantees, the purchasing process is a lot easier. Second, consolidated efforts like UCS and Vblocks are about software as much as they are about hardware. The promise of a true single pane of glass is very appealing.

If you look at some of the rebels in the software world, like Automattic and Flickr, you’ll notice something: their openness. In both of these cases they allow customers to do whatever they want with their own data. Want to pick it up and leave? That’s fine by them. Contrary to the thinking behind the sales models of EMC and Cisco, though, those companies have great customer retention rates. Because the fundamental idea that their customers are free to choose colors everything they do, they spend their time looking to make customers happy, not lock them in. And as a result you get products people want to use, and not products that are just a little less evil than the other guy.

Seeing that a true rebel alliance is unlikely, I’d love to see Dell become the open, rebel alternative to companies like EMC, Cisco, and Oracle. Even going so far as to develop a “Dblock” based on open technology, sold in increments, delivered in 19″ racks, managed through a single pane of glass, and marketed as the antithesis to vendor lock in. A Dblock could have Dell servers in it, Equallogic/3PAR storage, and Xsigo interconnects (imagine an array full of SSD and quad-rate Infiniband connections), and be very compelling. Especially if their answer to a question like “Can I attach my NetApp to it?” was “the ports are on the top left.” I’d be happy to do business with a company like that.

It’d just need one little twist: a change in attitude.

Over the last ten years my organization has come a long way with its IT policies and processes. We’ve gone from the wild, wild west of IT where personal heroism ruled the day, to a place where there’s just enough process to make sure that communication happens correctly and things like our Configuration Management Database (CMDB) stay up to date. It’s been a lot of work, but I am actually really proud of where we’re at.

There are three fundamental decisions we made a long time ago that, had they not been made, would have drastically changed how virtualization has proceeded here.

1. Clearly defined maintenance windows.

Knowing exactly when someone can do maintenance on server has been crucial to getting things done in our virtualization environment. There are many adjustments you can & should make in virtual environments, but if you can’t ever take the VMs down to make the changes you’re stuck. We’ve been able to do physical to virtual migrations, performance tuning, VMware Tools upgrades, vSphere upgrades, and a whole slew of other things in relatively short timeframes because we have this all worked out already. This also lets us “right-size” our VMs — rather than deploying huge VMs just in case they need the CPU or RAM, we deploy smaller ones and then can take an outage to add CPUs and RAM if we need to. The maintenance windows for a server are negotiated between the application/service admins and the system administrators when a machine is put into production, we track it in our CMDB, and any member of the whole team supporting the service can take the maintenance window, as long as they follow some rules about notifications for the change (timeframes, etc.).

2. Use of load-balancing technologies.

We use application load balancers (layer 4 of the OSI model) to decouple services from individual servers. Not only does this allow us to take a host down without affecting a service, but it also lets us spread the load out more among the physical hosts we have in our virtual infrastructure. In a lot of cases having more, smaller VMs results in better workload scheduling by ESX and DRS, especially on smaller ESX hosts.

Of course, this also plays nicely into the other points, because it’s very liberating to be able to do what we call “rolling maintenance” on a service, just taking one machine down at a time so that customers are not impacted. It also means that system administrator quality of life goes up, for now we can do maintenance tasks during the day instead of on weekends and off-hours. Doing maintenance during business hours has a couple of benefits. First, it means that the maintenance will actually get done. If you try to use someone’s personal time to do work they tend to opt out of that work. Servers go unpatched, tuning doesn’t happen, lots of things that should get done don’t because people will choose their personal time over work. Second, it means that if something goes wrong there are others around to help out. Doing work at 5 AM on a Sunday is fun, but if things go sideways you have to wake someone up or try fixing it yourself. Doing work during the day means you have the rest of the team around to lend a hand.

Third, it gives you a way to make incremental changes and then watch the effects. This has been particularly awesome for performance tuning of applications and our virtual environments themselves. Testing tuning changes is often hard, because test suites and test load generators are synthetic and often don’t compare to real load. But because the load is spread out we can make a change to one VM, or one ESX host servicing one VM, and keep an eye on it. I’m not advocating being a complete cowboy — you still have to do testing — but the risks to your production environment are a lot lower if you can catch problems on one VM first.

There are usually some other benefits to load balancers, too, that make them virtualization-friendly. Many will offload SSL processing, so your VMs have less work to do. Others have features, like iRules in F5′s products, that let you rewrite network traffic on the fly, which has some really neat implications for security, monitoring, and service delivery. And if you don’t want to buy a piece of hardware you can often get a virtual appliance from these vendors, though the physical appliances are usually a lot faster.

3. Commitment to operating system and application patching.

It is a fundamental belief of mine that one of the best ways to stay secure is to keep up on your patching. My organization agrees, and by using load balancers and defining maintenance windows we’ve made it easy for ourselves to keep our hosts up to date with regular patching cycles. Because we can take servers down without taking services down, and because sysadmins know exactly when a server can come down, we can schedule maintenance cycles easily, whether it’s six months out or two weeks. We can also respond very rapidly to emergency situations, like recent remote execution vulnerabilities in Microsoft Windows, by rolling patches out to development & test hosts, then QA & production, over the course of just two days if needed.

Keeping up to date with patches not only keeps you secure, it also lets you take advantage of new features that are added to operating systems. For example, Red Hat keeps adding new virtualization-friendly features, like kernel interrupt clock dividers. Being a kernel parameter you can’t just change it on the fly. And if you have to reboot, but can’t get a time to do it, you won’t do it. For us, we just rolled the change into one of our patching cycles and reduced the load on our infrastructure dramatically. Meaning more VMs per physical host, and a quantifiable amount of savings from just a small change on each machine.

Furthermore, our commitment to patching also extends to the virtual infrastructure itself, and we have a rule that we will not implement anything that breaks vMotion or Storage vMotion. Why? Because then it becomes very difficult to cope with ESX updates, or hardware failures, or any situation where vMotion could be used to prevent an outage. Sure, this means that we still need physical hardware for some applications, but it’s still just a fraction of the hardware we were buying years ago. This also makes virtual infrastructure easy to upgrade when the time comes, for new versions of vSphere, new storage arrays, and new physical hosts. Instead of planning outages on hundreds of VMs we just vMotion them, and nobody is the wiser.

Disclosure: F5 is a sponsor of Gestalt IT Tech Field Day, of which I have been a participant. I am not a customer of F5 at this time, though.

If you were thinking about going to VMworld but can’t make it due to finances Gestalt IT is running a contest where the winner gets airfare, hotel, and a conference pass. This is a very compelling contest, and to win:

Entrants must explain how they plan to “pay it forward” if they get to go to VMworld. Will you start a blog? Write some tutorials? Contribute to a forum or online community? Present to your local VMUG? Get creative and spread the wealth of knowledge you get from the event!

Our panel of judges is made up of none other than the most-excellent roster of past Tech Field Day delegates! They’ve proven themselves to be independent-minded and knowledgeable, and we’re sure that they will pick the best entry!

I’ll be one of the judges — I’m looking forward to seeing some really great proposals! Go over to Gestalt IT for all the details.

VMware vSphere users should always remember that when you allocate a VM the amount of space it will consume on disk includes a swap file equal to the size of the VM’s allocated RAM.

So if you have 96 GB of VMs running you will use 96 GB of your disk as swap, even if those VMs are only actively using 2 GB of RAM.

Yet another argument against overcommitment, in my opinion. If you right-size your VMs you not only save RAM, but storage as well.

Beth Pariseau from SearchServerVirtualization.com has a new piece up about the change in VMware licensing for some products, from per-socket to per-VM.

First off, and slightly off-topic, I’m quoted in the article, and it’s always interesting to see what reporters choose for quotes from me. This isn’t a criticism, as anybody who has talked to me in real life knows I talk like that. Beth didn’t edit anything, which I really appreciate. However, I now can cross “use the word ‘cram’ in an interview” off my list of life goals. :)

Second, and back on-topic, per-VM licensing makes a lot of sense for things like AppSpeed, as well as some of the other management tools. There are a lot of VMs in my environments where I don’t need to know application latency, for example, and I’d rather not pay for for licenses for those VMs. I know just a little about Site Recovery Manager, but if this license model is similar that would be great for companies that don’t need to protect all of their VMs.

On the other hand, per-VM licensing for a tool like CapacityIQ is pretty sketchy, in my opinion. You are trying to use that tool to increase the number of VMs in your environment, which in turn increases your licensing fees. That’s counterproductive, and I can see how a number of potential licensees may be turned off by this change, at least specifically for CapacityIQ.

Changing a licensing model isn’t ever easy. A lot of IT shops are the least nimble business units you’ll ever see, and any change to anything evokes Chicken Little-esque comments, complete with falling skies. Certainly with a change like this comes a little more management overhead, because you have to manage two licensing schemes. But virtualization is about driving costs out of IT, plain and simple, and these changes are likely to outweigh the cost of staff time needed to cope with them. They’ll also open the door for some customers who were previously put off by the high prices. I think that’s a win.

Once again we find ourselves staring at a major release of VMware infrastructure software: vSphere 4.1. It’s been a bit over a year since 4.0 dropped, with two big bugfix releases since. vSphere 4.1 adds over 150 new features and improvements, including some features that were previewed at VMworld 2009 to much applause. Here are some of the highlights, twists, and turns.

Storage I/O Control:

This is a global, cluster-wide I/O scheduler, working to throttle I/O to ensure that a single VM cannot monopolize a single datastore’s capabilities. If you consider that a datastore’s backend storage can only sustain a certain numbers of IOPS it’s possible, and likely, that a single VM will consume a disproportionate amount of those IOPS. This feature lets you control which VMs get priority when there is congestion.

This is enabled per-datastore, and will kick in when latencies are 30 ms or greater by default, though you can adjust the latencies as you see fit via advanced settings. This feature is configured through VMware vCenter, but the ESX servers communicate between themselves via shared headers on each datastore. This preserves the ability for the cluster to continue to operate when the management interfaces are inoperative.

Storage I/O Control is only available to Enterprise Plus customers.

Network I/O Control:

As network connectivity on hosts moves from multiple 1 Gbps Ethernet links towards one or two 10 Gbps per host it becomes more important that network operations get prioritized, to avoid starving any particular operation. Network I/O Control does this by adding a few different features.

First, it implements the ability to schedule and prioritize traffic types. There are six types of traffic that are identified: VM, management, NFS, iSCSI, VMotion, and FT logging. You can assign limits to these types of traffic, to absolutely cap the amount of bandwidth being consumed by each. You can also assign shares to these types of traffic to designate the relative priority of that traffic. These limits and shares are always enacted on egress, meaning that it’s only outbound traffic that is throttled.

Second, a new feature called Load-Based Teaming has been implemented, to alter the physical NIC being used for a type of traffic when ESX detects that a link is more than 75% utilized over 30 seconds. This will be especially useful for 1 Gbps links, spreading the load across different links as utilization rises.

This technology only works as part of a vNetwork Distributed Switch, and as such is only part of Enterprise Plus licenses.

Memory Compression:

Memory overcommitment, while a hotly-debated topic at times, is a reality for a lot of people. Overcommitment is a gamble, though, and sometimes you end up with virtual machines that, in total, need more RAM than you have. vSphere 4 tried helping you out in three ways when you were caught short. First, it used transparent page sharing to try to deduplicate RAM. Second, it used the VMware Tools balloon driver to reduce the amount of RAM actively consumed by a VM. Third, it just paged to disk.

Paging to disk is a terribly slow operation. Disks operate thousands to millions of times slower than RAM  and CPUs, so VMware added another technology to stave off the need to page to disk: memory compression. ESX will take a configurable amount of memory and compress it. Of course, the optimal situation is to not need any of these technologies, but another way to avoid disk accesses is great.

Those of you having bad flashbacks to MS-DOS and RAM Doubler should consider that CPUs and RAM are much faster, this is configurable via the advanced settings on a per-VM basis, and that VMware says that decompression takes about 20 μs, which is still way faster than disk.

Memory compression is available to all licensed customers.

vMotion & EVC Improvements:

vMotion has had evolutionary improvements to it, to significantly decrease migration times. They do this through some behind-the-scenes improvements, as well as increasing the number of concurrent vMotions. You can now have 4 vMotions running simultaneously on a 1 Gbps link, and 8 on a 10 Gbps link. One datastore can also have 128 concurrent vMotions running on it.

Enhanced vMotion Compatibility (EVC) has also been updated to consider the CPU feature set of a running VM, rather than the features supported by the host. This allows for some better compatibility while moving VMs around, and better error detection when adding new hosts to clusters.

vMotion is also now officially spelled with a lowercase ‘v’ — whew! :)

vStorage API for Array Integration (VAAI):

Any old mainframe IT guy will look at all this new virtualization technology and make a snide remark that they had all this 25 years ago. In a lot of cases it’s true, especially when you consider coprocessors. While my iPhone has more CPU time than a mid-1990s mainframe, the mainframe didn’t actually have to use its CPU for much. Everything non-CPU was offloaded to coprocessors, especially I/O.

Continuing this trend among us naive open systems types, VMware and a variety of storage partners are working to enable offloading of storage operations to the arrays themselves, coprocessor-style. Initially there will be three operations handled by the arrays: full copy, zero-out, and locking. For operations like cloning this means that ESX won’t copy a template out over the SAN or network just to put it right back on the array. This is going to be a killer performance improvement for people using slower storage link, like iSCSI over 1 Gbps links. Likewise, the locking functions will help avoid locking the whole datastore for management operations. That will make datastores more scalable, as locking is a concern for big datastores.

Actual support on arrays is forthcoming, and vendors such as Dell, NetApp, IBM, Hitachi, HP, and EMC are actively working to release compatible array firmware. This feature is only available to Enterprise and Enterprise Plus licensees.

ESXi:

vSphere 4.1 is the last major release to support the classic ESX software with the service console. Future releases will be ESXi only, using APIs to control and configure the host, and people should consider moving towards ESXi. There is a nice “VMware ESX to ESXi Upgrade Center” to answer a lot of questions, and help smooth the transition.

ESX 4.1 really brings it up to the level that ESX has been at for a while. They have added boot-from-SAN capabilities, scripted installs, enhanced Update Manager to push drivers and other modules, added built-in Active Directory support, and now fully support both local and remote Tech Support Mode.

This will be a big change for a lot of people, and some thought and testing would be prudent for everybody, beginning now.

High Availability:

High Availability has had its limits increased, to 32 hosts, 320 VMs per host, and 3000 VMs per cluster. These limits are enforced after a failover, of course, so you still need to think about your cluster as N+1, and keep N under those limits. The concept/limit of 5 primary nodes still exists as it was.

Reporting of problems is much improved, via the new Cluster Operational Status window. It shows a better view of what is wrong, as determined by a new health check process. High Availability and DRS now work together to move resources to better help restart VMs after a failure.

High Availability now also has APIs for working with applications. This lets monitoring agents work with HA to do a variety of things, including a full guest restart. This works through the VMware Tools and involves guest to host communication, which may be a security concern in some cases. However, while VMware hasn’t said much so far it opens the door for some interesting monitoring and self-healing possibilities.

Fault Tolerance (FT):

Still no SMP support in FT — that’s a much harder problem than it seems because there’s hardware support for FT for single-CPU VMs, but none for SMP VMs. However, a lot of other restrictions have been lifted. DRS can now move FT VMs around, which is great. Having FT VMs locked to a specific host was terribly restricting. It requires having EVC enabled, but a lot of people are already doing that. You can also have ESX hosts at different versions.

Dynamic Resource Scheduling & Dynamic Power Management:

There is a new set of host affinity rules for DRS, that can give VMs affinity (or anti-affinity) for a set of hosts in a cluster. This helps a lot when someone is faced with host-based licensing restrictions, for instance. These rules can be set to preferred or required, and both DRS and HA will obey these rules.

DPM now has a set of scheduled tasks to help control it, turning it on and off at certain times of the day if you’d like. Disabling DPM will bring all the hosts out of standby, to help guarantee that no hosts get stuck in a useless state.

vCenter:

The changes to vCenter are really too numerous to list here. The biggest change is that it is 64-bit only, which will be a giant problem for a lot of people who are running 32-bit Windows. There are a lot of other incremental improvements, including a lot more information being presented to the users, often in conjunction with the new features throughout vSphere 4.1.

Whew. Get testing, people!

I’ve had occasion, over the last 24 hours, to change the logging level of my vCenter server. It’s been a little more crashy than usual, and I’ve discovered that a number of KB articles are not updated for vSphere 4.0 Update 2 or Windows Server 2008.

If you want to change the log retention policies, you can follow most of the instructions in KB 1004795 but there are two things I think you should know if you’re running on Windows 2008:

1. Your vpxd.cfg file and your logs are actually in:

C:\ProgramData\VMware\VMware VirtualCenter

2. The XML stanza for the logging is commented out in 4.0 Update 2, so remove the <!– and –> lines to make it work. For example, make:

...
  <logName>CpuFeatures</logName>
 </level>
 <!--
 <log>
   <maxFileNum>10</maxFileNum>
   <compressOnRoll>false</compressOnRoll>
   <level>info</level>
 </log>
 -->
 <vpxd>
    <das>
...

into:

...
   <logName>CpuFeatures</logName>
 </level>
 <log>
   <maxFileNum>100</maxFileNum>
   <compressOnRoll>true</compressOnRoll>
   <level>info</level>
 </log>
 <vpxd>
    <das>
...

You may also want to change the compressOnRoll to ‘true’ as I did above if you are concerned about disk space usage.

This may seem obvious to a lot of people, but for those that only occasionally make changes to vpxd.cfg it helps to have been pre-warned about the differences. At least you’ll vaguely remember that something’s changed… :)

If you’ve added a volume to a VMware Data Recovery appliance you’ve probably read the part of the Administration Guide entitled “Add a Hard Disk to the Backup Appliance,” especially page 18 where it says:

If creating a SCSI virtual disk, it is recommended that you set the SCSI value to SCSI 1:0.

When you specify a SCSI Virtual Device Node the numbering scheme is Bus:ID, so what they’re really recommending by suggesting 1:0 is to let vSphere create another virtual SCSI controller. By default, that new SCSI controller will be set to “LSI Logic Parallel.”

What I’ve noticed, though, is that the VMware Data Recovery appliance has the pvscsi modules installed, so you can change the type of SCSI controller to “Paravirtual” and take advantage of the performance gains. Nothing is mentioned in the documentation, pro or con, but it’s been working for me under VDR 1.1 and now under VDR 1.2. I’ll certainly take the free performance boost, especially on an I/O intensive VM!