The Lone Sysadmin » Featured

VMworld 2010: Hands-On Labs

Bob Plankers — Sun, 29 Aug 2010 21:44:51 +0000

This morning John Troyer coordinated a bunch of bloggers for a session over at the VMworld 2010 Hands-On Lab facilities in Moscone West. Adam Zimman, Dan Anderson, and Curtis Pope took turns explaining and demoing the lab to us. The lab itself was built as a cloud-oriented system, using software-on-demand and service-on-demand principles, and relying heavily on remotely-hosted equipment in data centers in Miami, FL (Terremark) and Ashburn, VA (Verizon).

The Lab team is really building on what they’ve learned from other years. There are many more labs this year than last, and they’re all self-paced, though there are options for instructor interaction as well if you have questions or want more one-on-one guidance. Self-paced labs means they can do almost unlimited content, and it’s easier to get lots of people through the labs. Last year they had, all totalled, about 7000 lab seat hours. This year they have almost 20,000, with 480 View stations in eight rooms. Dan Anderson, the lab’s lead architect, had some proud things to say about what they’ve done. “The content is killer, the best content I’ve seen yet. If someone sits for four days, eight hours a day, they might be able to get through all of them. But nobody can complain about not having enough stick time,” said Dan.

Perhaps he’s never met some of the curmudgeonly people that attend VMworld. :) But I really appreciate the iterative approach they’ve taken this year to making the labs better. For instance, they learned that pre-registration for the labs didn’t work very well in other years, so it’s all first-come, first-served (FIFO). There’s a check-in station that works with your badge number, and a waiting room with couches and whiteboards and Subject Matter Experts while you wait. The labs will be open from 8 AM until 10 PM every day, too, and they will be offering a prize to “dedicated individuals” (they thought speed and quantity might be the factors, but it isn’t set in stone). They did say the prizes would be something like a pass to VMworld 2011, though, which is very cool.

The hardware and software powering the lab is pretty amazing, with a number of sponsors contributing staff, equipment, and software to make it run. Sometimes on very short notice, too. And in some cases this lab is the largest deployment yet of these technologies. They’re pre-populating lab environments with instances of each lab setup, to avoid the on-demand 5 to 7 minute wait from last year, which is great. They’re worried that they’ll have the prepopulation levels off a little on the first day, but even if you do get caught waiting you can still read the manuals. They estimate that the labs are using roughly 36 TB of RAM (yes, TB) and there’s about 200 TB of storage, between EMC and NetApp, in each data center powering the labs, all connected via NFS. The storage itself is everything from enterprise flash (EFD) to SATA, with the EFD often being used as FastCache to front-end the slower storage.

The stations themselves are Wyse thin clients, with dual monitors and even dual chairs, even though it’s geared for one-on-one learning. It’s all about flexibility and options, which extends to the content itself — the vSphere Sandbox lab is just a deployment of all of their products, for freeform messing around. The Lab team even has redundant wiring to the lab stations, just in case they need it (“We even have redundant chairs!” said Adam). They’re flexible, they’re ready, and they’re hoping that they can set records for the number of happy people in the labs this year. And if you’re not happy, there’s 150 staff floating around to help you out, as well as two lab captains per room.

I’m looking forward to it — labs have always been a highlight of VMworld for me, and these guys are making it even better. I know it’s a lot of work to build, in two months, what usually would be done in a year or two (and then, as Dan said, “throw it on a truck.”). On behalf of all of us, thank you Adam, Dan, and Curtis (and all the others that we didn’t meet). I hope all your hard work is a giant success!

VMworld 2010: Saturday

Bob Plankers — Sun, 29 Aug 2010 16:25:32 +0000

A quick walk past Moscone on Saturday yielded a real life “401: Not Authorized,” most likely because it’s “501: Not Implemented” yet. So we continued on down to AT&T Park to watch the Giants lose to the Diamondbacks, 11-3 (it was 6-1 after the first inning, Barry Zito was having a real bad night). At least we got Joe DiMaggio bobbleheads. Not much else to do, it looks like a bunch of folks led by the inimitable Veeam guys went to the Chieftain. My coworker Steve and I opted for an early turn-in, to hedge against the upcoming late ones. Below are some photos.

On tap for Sunday: a sneak preview of the Labs this morning, brunch with a bunch of the Communities guys, registration and speaker setup at 2 PM, and then nothing. The vmunderground.com WUPaaS is tonight, but I missed the invites, so I might try sneaking in. Otherwise I’m sure other things will be going on. I do need to work on my presentation a bit, and perhaps a trip over into the Sunset district for some excellent Vietnamese at Pho Phu Quoc is in order, followed by a walk down Irving to the beach and the N Judah home.

I hope you’re all traveling well. Stay safe, and remember that if you’ve got a question or are looking for something to do #vmworld is a good start on Twitter. You can also message me, @plankers.

Rebel Alliances

Bob Plankers — Thu, 26 Aug 2010 19:06:45 +0000

Stephen Foskett has a great post today on The Enterprise IT Acquisition Game, wherein he talks about how it’s open season on data storage companies, with a lesser emphasis on networking:

So this is the game: Four full-line enterprise superpowers battling each other for datacenter dominance and coveting the extra profits of a few verticals. HP clearly believes they can chip away at EMC and Cisco in storage and networking; Dell and IBM have so far focused mainly on storage; and Oracle hasn’t made a move in either direction, instead challenging the other three in the core server and software space.

Right on, especially with the “coveting the extra profits” part. For years, Dell, IBM, and HP have been busy commoditizing the compute node side of things. They’ve been driving the prices down on CPU and RAM for so long that the margins aren’t there anymore. And now, with the onset of widespread virtualization, the volume is no longer there to make up for it. However, storage is still a high-margin endeavor, and probably the single most expensive thing in my data center; I have $400K in CPU & RAM connected to $2m in storage. It is not surprising that there is a huge bidding war for 3PAR, and an open season on data storage companies. It’s one of the best ways for big companies to continue bleeding IT budgets, at least until SSD capacities rise and prices fall, and it does to storage what virtualization is doing to servers.

The thing is, virtualization is all about driving costs down in IT, and I just don’t see these big consolidation efforts as an actual way to do that in the long term. EMC has been downright nasty with my organization because they thought we were locked in. Cisco has done the same, and Oracle is generally regarded a bunch of jerks. Why would anybody choose UCS, VBlocks, or an all-in-one option from Oracle knowing that you’re locking yourself in? Even if the pricing is competitive right now, it won’t be for the fourth or fifth year of support where they think they’ve got you trapped.

Stephen’s last paragraph speaks of rebel alliances:

Although I would love to see a rebel alliance rise (imagine Juniper, NetApp, and Symantec joining forces!) this is not a likely scenario.

Like Stephen, I would enjoy seeing a rebel alliance, but I doubt it is likely, for two reasons. One, companies like buying from a single source, because they can get support guarantees, the purchasing process is a lot easier. Second, consolidated efforts like UCS and Vblocks are about software as much as they are about hardware. The promise of a true single pane of glass is very appealing.

If you look at some of the rebels in the software world, like Automattic and Flickr, you’ll notice something: their openness. In both of these cases they allow customers to do whatever they want with their own data. Want to pick it up and leave? That’s fine by them. Contrary to the thinking behind the sales models of EMC and Cisco, though, those companies have great customer retention rates. Because the fundamental idea that their customers are free to choose colors everything they do, they spend their time looking to make customers happy, not lock them in. And as a result you get products people want to use, and not products that are just a little less evil than the other guy.

Seeing that a true rebel alliance is unlikely, I’d love to see Dell become the open, rebel alternative to companies like EMC, Cisco, and Oracle. Even going so far as to develop a “Dblock” based on open technology, sold in increments, delivered in 19″ racks, managed through a single pane of glass, and marketed as the antithesis to vendor lock in. A Dblock could have Dell servers in it, Equallogic/3PAR storage, and Xsigo interconnects (imagine an array full of SSD and quad-rate Infiniband connections), and be very compelling. Especially if their answer to a question like “Can I attach my NetApp to it?” was “the ports are on the top left.” I’d be happy to do business with a company like that.

It’d just need one little twist: a change in attitude.

Three Organizational Decisions That Help Me Virtualize

Bob Plankers — Fri, 06 Aug 2010 18:55:55 +0000

Over the last ten years my organization has come a long way with its IT policies and processes. We’ve gone from the wild, wild west of IT where personal heroism ruled the day, to a place where there’s just enough process to make sure that communication happens correctly and things like our Configuration Management Database (CMDB) stay up to date. It’s been a lot of work, but I am actually really proud of where we’re at.

There are three fundamental decisions we made a long time ago that, had they not been made, would have drastically changed how virtualization has proceeded here.

1. Clearly defined maintenance windows.

Knowing exactly when someone can do maintenance on server has been crucial to getting things done in our virtualization environment. There are many adjustments you can & should make in virtual environments, but if you can’t ever take the VMs down to make the changes you’re stuck. We’ve been able to do physical to virtual migrations, performance tuning, VMware Tools upgrades, vSphere upgrades, and a whole slew of other things in relatively short timeframes because we have this all worked out already. This also lets us “right-size” our VMs — rather than deploying huge VMs just in case they need the CPU or RAM, we deploy smaller ones and then can take an outage to add CPUs and RAM if we need to. The maintenance windows for a server are negotiated between the application/service admins and the system administrators when a machine is put into production, we track it in our CMDB, and any member of the whole team supporting the service can take the maintenance window, as long as they follow some rules about notifications for the change (timeframes, etc.).

2. Use of load-balancing technologies.

We use application load balancers (layer 4 of the OSI model) to decouple services from individual servers. Not only does this allow us to take a host down without affecting a service, but it also lets us spread the load out more among the physical hosts we have in our virtual infrastructure. In a lot of cases having more, smaller VMs results in better workload scheduling by ESX and DRS, especially on smaller ESX hosts.

Of course, this also plays nicely into the other points, because it’s very liberating to be able to do what we call “rolling maintenance” on a service, just taking one machine down at a time so that customers are not impacted. It also means that system administrator quality of life goes up, for now we can do maintenance tasks during the day instead of on weekends and off-hours. Doing maintenance during business hours has a couple of benefits. First, it means that the maintenance will actually get done. If you try to use someone’s personal time to do work they tend to opt out of that work. Servers go unpatched, tuning doesn’t happen, lots of things that should get done don’t because people will choose their personal time over work. Second, it means that if something goes wrong there are others around to help out. Doing work at 5 AM on a Sunday is fun, but if things go sideways you have to wake someone up or try fixing it yourself. Doing work during the day means you have the rest of the team around to lend a hand.

Third, it gives you a way to make incremental changes and then watch the effects. This has been particularly awesome for performance tuning of applications and our virtual environments themselves. Testing tuning changes is often hard, because test suites and test load generators are synthetic and often don’t compare to real load. But because the load is spread out we can make a change to one VM, or one ESX host servicing one VM, and keep an eye on it. I’m not advocating being a complete cowboy — you still have to do testing — but the risks to your production environment are a lot lower if you can catch problems on one VM first.

There are usually some other benefits to load balancers, too, that make them virtualization-friendly. Many will offload SSL processing, so your VMs have less work to do. Others have features, like iRules in F5′s products, that let you rewrite network traffic on the fly, which has some really neat implications for security, monitoring, and service delivery. And if you don’t want to buy a piece of hardware you can often get a virtual appliance from these vendors, though the physical appliances are usually a lot faster.

3. Commitment to operating system and application patching.

It is a fundamental belief of mine that one of the best ways to stay secure is to keep up on your patching. My organization agrees, and by using load balancers and defining maintenance windows we’ve made it easy for ourselves to keep our hosts up to date with regular patching cycles. Because we can take servers down without taking services down, and because sysadmins know exactly when a server can come down, we can schedule maintenance cycles easily, whether it’s six months out or two weeks. We can also respond very rapidly to emergency situations, like recent remote execution vulnerabilities in Microsoft Windows, by rolling patches out to development & test hosts, then QA & production, over the course of just two days if needed.

Keeping up to date with patches not only keeps you secure, it also lets you take advantage of new features that are added to operating systems. For example, Red Hat keeps adding new virtualization-friendly features, like kernel interrupt clock dividers. Being a kernel parameter you can’t just change it on the fly. And if you have to reboot, but can’t get a time to do it, you won’t do it. For us, we just rolled the change into one of our patching cycles and reduced the load on our infrastructure dramatically. Meaning more VMs per physical host, and a quantifiable amount of savings from just a small change on each machine.

Furthermore, our commitment to patching also extends to the virtual infrastructure itself, and we have a rule that we will not implement anything that breaks vMotion or Storage vMotion. Why? Because then it becomes very difficult to cope with ESX updates, or hardware failures, or any situation where vMotion could be used to prevent an outage. Sure, this means that we still need physical hardware for some applications, but it’s still just a fraction of the hardware we were buying years ago. This also makes virtual infrastructure easy to upgrade when the time comes, for new versions of vSphere, new storage arrays, and new physical hosts. Instead of planning outages on hundreds of VMs we just vMotion them, and nobody is the wiser.

Disclosure: F5 is a sponsor of Gestalt IT Tech Field Day, of which I have been a participant. I am not a customer of F5 at this time, though.

Youth

Bob Plankers — Thu, 05 Aug 2010 23:45:09 +0000

“Hey, do any of you guys have an old, full-height hard disk lying around?”

This was a relatively new person from another group in our organization. People occasionally come looking for random old equipment to use for training & examples, because they know we have things like original IBM PCs, Cisco AGS+ routers, token ring MAUs, and 1200 baud Multitech Multimodems on hand.

“Sure, I’ve got a full height drive, one second.” I produce a full-height 600 MB Imprimis SCSI disk. Made in the USA, so it’s pretty old. It’s a bookend on my bookshelf.

“What in the heck is that?” he asks.

“Um, a full-height drive?” I reply, really wondering what he thinks he’s asking for.

“No, man, I don’t know what that’s out of but it’s wicked. Full-height is like a couple inches tall, though.”

“And 3.5″ form factor, right?”

“Yeah.”

“Dude, that’s half height.”

“Nah, that’s full height, at least that’s what I’ve been told. So what is that?” he asks, as he points to my impressive specimen of early 1990s drive technology.

“What you were looking for is half height. This is a full height 600 MB SCSI fixed disk. Final answer.” I hope he didn’t learn full height vs. half height from someone he paid.

“I’ve never seen one of those before. Can I borrow it? The other guys will flip out when they see this thing.”

I wonder what they’d think if they saw 8″ floppy disks. Freakin’ kids.

Get Away to VMworld Contest

Bob Plankers — Wed, 04 Aug 2010 18:28:58 +0000

If you were thinking about going to VMworld but can’t make it due to finances Gestalt IT is running a contest where the winner gets airfare, hotel, and a conference pass. This is a very compelling contest, and to win:

Entrants must explain how they plan to “pay it forward” if they get to go to VMworld. Will you start a blog? Write some tutorials? Contribute to a forum or online community? Present to your local VMUG? Get creative and spread the wealth of knowledge you get from the event!

Our panel of judges is made up of none other than the most-excellent roster of past Tech Field Day delegates! They’ve proven themselves to be independent-minded and knowledgeable, and we’re sure that they will pick the best entry!

I’ll be one of the judges — I’m looking forward to seeing some really great proposals! Go over to Gestalt IT for all the details.

Oregon Trail: The Movie

Bob Plankers — Mon, 02 Aug 2010 19:43:32 +0000

One would think this is off-topic, but for techies it really isn’t. I present to you the official trailer for The Oregon Trail. Very well done!

Makes me proud to have grown up in Minnesota (MECC == “Minnesota Educational Computing Consortium”). If you want to play it Classic Gaming has it and a copy of AppleWin.

Remembering Swap

Bob Plankers — Mon, 02 Aug 2010 15:02:59 +0000

VMware vSphere users should always remember that when you allocate a VM the amount of space it will consume on disk includes a swap file equal to the size of the VM’s allocated RAM.

So if you have 96 GB of VMs running you will use 96 GB of your disk as swap, even if those VMs are only actively using 2 GB of RAM.

Yet another argument against overcommitment, in my opinion. If you right-size your VMs you not only save RAM, but storage as well.

Happy System Administrator Appreciation Day!

Bob Plankers — Fri, 30 Jul 2010 17:01:06 +0000

The Wisconsin DMV sent me my gift a day early:

And it was a present — I needed replacement plates but hadn’t ordered them yet. I’m glad I didn’t!

I often joke that I haven’t come up with an original solution to anything in years, thanks to all the other sysadmins out there who share their solutions, knowledge, and time in order to make the world better. Thank you all for everything you do!

Rate-Limiting Steps

Bob Plankers — Mon, 26 Jul 2010 18:02:14 +0000

In the last month I’ve added quite a few blogs to my reading list. One new one is “Movin’ Meat,” written by an ER doctor out of the Pacific Northwest. Besides just being interesting, some of his blog posts support my theory that IT folks can often learn things from people in other fields. The post from June 25, 2010, part four of his “Advice for Interns,” is one of these cases. When you read it (link is at the end because I want to get to my actual point before you leave to read it), I think substituting “customer/system” for “patient” in his list works nicely.

My real point is this: one thing in his list really stood out for me. It’s something that seems really obvious when it’s said, but also done wrong a lot:

“Determine the rate-limiting step and make it priority #1 in the work-up”

Figuring out what the slowest step in a project is going to be and getting to work on it right away is often key to getting a project done in a timely fashion. Especially if a large chunk of that time will be waiting for something. When you know it’s going to take six weeks for a request to make it through your purchasing department you should start that right away, especially since all you’ll be doing is waiting.

As kids we were told by our teachers to read all the instructions first, then start working on whatever we were doing. Determining the rate-limiting steps is the same sort of thing. By taking a few minutes at the beginning to look at the whole project first, rather than just starting on step #1 and going one by one until you’re done, you can often optimize things so that the longest parts of the project are done in parallel with the rest.

Links:

- Movin’ Meat: Friday Flashback – Advice for Interns Part Four

Per-VM Licensing

Bob Plankers — Fri, 23 Jul 2010 17:42:00 +0000

Beth Pariseau from SearchServerVirtualization.com has a new piece up about the change in VMware licensing for some products, from per-socket to per-VM.

First off, and slightly off-topic, I’m quoted in the article, and it’s always interesting to see what reporters choose for quotes from me. This isn’t a criticism, as anybody who has talked to me in real life knows I talk like that. Beth didn’t edit anything, which I really appreciate. However, I now can cross “use the word ‘cram’ in an interview” off my list of life goals. :)

Second, and back on-topic, per-VM licensing makes a lot of sense for things like AppSpeed, as well as some of the other management tools. There are a lot of VMs in my environments where I don’t need to know application latency, for example, and I’d rather not pay for for licenses for those VMs. I know just a little about Site Recovery Manager, but if this license model is similar that would be great for companies that don’t need to protect all of their VMs.

On the other hand, per-VM licensing for a tool like CapacityIQ is pretty sketchy, in my opinion. You are trying to use that tool to increase the number of VMs in your environment, which in turn increases your licensing fees. That’s counterproductive, and I can see how a number of potential licensees may be turned off by this change, at least specifically for CapacityIQ.

Changing a licensing model isn’t ever easy. A lot of IT shops are the least nimble business units you’ll ever see, and any change to anything evokes Chicken Little-esque comments, complete with falling skies. Certainly with a change like this comes a little more management overhead, because you have to manage two licensing schemes. But virtualization is about driving costs out of IT, plain and simple, and these changes are likely to outweigh the cost of staff time needed to cope with them. They’ll also open the door for some customers who were previously put off by the high prices. I think that’s a win.

Gestalt IT Tech Field Day Seattle

Bob Plankers — Wed, 14 Jul 2010 22:01:06 +0000

Apple iPad: $670

Wyse PocketCloud RDP/View Client: $14.99

One flight worth of GoGoWireless: $12.95

Posting on my blog via an RDP connection to my work desktop across a VPN from 30,000 feet: priceless.

I’m on my way to Gestalt IT’s Seattle Tech Field Day. I’m excited, for a lot of reasons. It’s an honor to be invited, nominated by some of the other delegates. I’ve spent little time in Seattle, and while I won’t have a lot of extra free time this trip it’ll be better than last time I was there. I’ll get to hang out at a bunch of high-tech places, and best yet, do so with a bunch of high-tech folks that, frankly, I’ve only read about. How cool is that?

Pretty damn cool, if you ask me. And you didn’t, I know. :)

VMware vSphere 4.1: What’s New

Bob Plankers — Tue, 13 Jul 2010 06:05:44 +0000

Once again we find ourselves staring at a major release of VMware infrastructure software: vSphere 4.1. It’s been a bit over a year since 4.0 dropped, with two big bugfix releases since. vSphere 4.1 adds over 150 new features and improvements, including some features that were previewed at VMworld 2009 to much applause. Here are some of the highlights, twists, and turns.

Storage I/O Control:

This is a global, cluster-wide I/O scheduler, working to throttle I/O to ensure that a single VM cannot monopolize a single datastore’s capabilities. If you consider that a datastore’s backend storage can only sustain a certain numbers of IOPS it’s possible, and likely, that a single VM will consume a disproportionate amount of those IOPS. This feature lets you control which VMs get priority when there is congestion.

This is enabled per-datastore, and will kick in when latencies are 30 ms or greater by default, though you can adjust the latencies as you see fit via advanced settings. This feature is configured through VMware vCenter, but the ESX servers communicate between themselves via shared headers on each datastore. This preserves the ability for the cluster to continue to operate when the management interfaces are inoperative.

Storage I/O Control is only available to Enterprise Plus customers.

Network I/O Control:

As network connectivity on hosts moves from multiple 1 Gbps Ethernet links towards one or two 10 Gbps per host it becomes more important that network operations get prioritized, to avoid starving any particular operation. Network I/O Control does this by adding a few different features.

First, it implements the ability to schedule and prioritize traffic types. There are six types of traffic that are identified: VM, management, NFS, iSCSI, VMotion, and FT logging. You can assign limits to these types of traffic, to absolutely cap the amount of bandwidth being consumed by each. You can also assign shares to these types of traffic to designate the relative priority of that traffic. These limits and shares are always enacted on egress, meaning that it’s only outbound traffic that is throttled.

Second, a new feature called Load-Based Teaming has been implemented, to alter the physical NIC being used for a type of traffic when ESX detects that a link is more than 75% utilized over 30 seconds. This will be especially useful for 1 Gbps links, spreading the load across different links as utilization rises.

This technology only works as part of a vNetwork Distributed Switch, and as such is only part of Enterprise Plus licenses.

Memory Compression:

Memory overcommitment, while a hotly-debated topic at times, is a reality for a lot of people. Overcommitment is a gamble, though, and sometimes you end up with virtual machines that, in total, need more RAM than you have. vSphere 4 tried helping you out in three ways when you were caught short. First, it used transparent page sharing to try to deduplicate RAM. Second, it used the VMware Tools balloon driver to reduce the amount of RAM actively consumed by a VM. Third, it just paged to disk.

Paging to disk is a terribly slow operation. Disks operate thousands to millions of times slower than RAM and CPUs, so VMware added another technology to stave off the need to page to disk: memory compression. ESX will take a configurable amount of memory and compress it. Of course, the optimal situation is to not need any of these technologies, but another way to avoid disk accesses is great.

Those of you having bad flashbacks to MS-DOS and RAM Doubler should consider that CPUs and RAM are much faster, this is configurable via the advanced settings on a per-VM basis, and that VMware says that decompression takes about 20 μs, which is still way faster than disk.

Memory compression is available to all licensed customers.

vMotion & EVC Improvements:

vMotion has had evolutionary improvements to it, to significantly decrease migration times. They do this through some behind-the-scenes improvements, as well as increasing the number of concurrent vMotions. You can now have 4 vMotions running simultaneously on a 1 Gbps link, and 8 on a 10 Gbps link. One datastore can also have 128 concurrent vMotions running on it.

Enhanced vMotion Compatibility (EVC) has also been updated to consider the CPU feature set of a running VM, rather than the features supported by the host. This allows for some better compatibility while moving VMs around, and better error detection when adding new hosts to clusters.

vMotion is also now officially spelled with a lowercase ‘v’ — whew! :)

vStorage API for Array Integration (VAAI):

Any old mainframe IT guy will look at all this new virtualization technology and make a snide remark that they had all this 25 years ago. In a lot of cases it’s true, especially when you consider coprocessors. While my iPhone has more CPU time than a mid-1990s mainframe, the mainframe didn’t actually have to use its CPU for much. Everything non-CPU was offloaded to coprocessors, especially I/O.

Continuing this trend among us naive open systems types, VMware and a variety of storage partners are working to enable offloading of storage operations to the arrays themselves, coprocessor-style. Initially there will be three operations handled by the arrays: full copy, zero-out, and locking. For operations like cloning this means that ESX won’t copy a template out over the SAN or network just to put it right back on the array. This is going to be a killer performance improvement for people using slower storage link, like iSCSI over 1 Gbps links. Likewise, the locking functions will help avoid locking the whole datastore for management operations. That will make datastores more scalable, as locking is a concern for big datastores.

Actual support on arrays is forthcoming, and vendors such as Dell, NetApp, IBM, Hitachi, HP, and EMC are actively working to release compatible array firmware. This feature is only available to Enterprise and Enterprise Plus licensees.

ESXi:

vSphere 4.1 is the last major release to support the classic ESX software with the service console. Future releases will be ESXi only, using APIs to control and configure the host, and people should consider moving towards ESXi. There is a nice “VMware ESX to ESXi Upgrade Center” to answer a lot of questions, and help smooth the transition.

ESX 4.1 really brings it up to the level that ESX has been at for a while. They have added boot-from-SAN capabilities, scripted installs, enhanced Update Manager to push drivers and other modules, added built-in Active Directory support, and now fully support both local and remote Tech Support Mode.

This will be a big change for a lot of people, and some thought and testing would be prudent for everybody, beginning now.

High Availability:

High Availability has had its limits increased, to 32 hosts, 320 VMs per host, and 3000 VMs per cluster. These limits are enforced after a failover, of course, so you still need to think about your cluster as N+1, and keep N under those limits. The concept/limit of 5 primary nodes still exists as it was.

Reporting of problems is much improved, via the new Cluster Operational Status window. It shows a better view of what is wrong, as determined by a new health check process. High Availability and DRS now work together to move resources to better help restart VMs after a failure.

High Availability now also has APIs for working with applications. This lets monitoring agents work with HA to do a variety of things, including a full guest restart. This works through the VMware Tools and involves guest to host communication, which may be a security concern in some cases. However, while VMware hasn’t said much so far it opens the door for some interesting monitoring and self-healing possibilities.

Fault Tolerance (FT):

Still no SMP support in FT — that’s a much harder problem than it seems because there’s hardware support for FT for single-CPU VMs, but none for SMP VMs. However, a lot of other restrictions have been lifted. DRS can now move FT VMs around, which is great. Having FT VMs locked to a specific host was terribly restricting. It requires having EVC enabled, but a lot of people are already doing that. You can also have ESX hosts at different versions.

Dynamic Resource Scheduling & Dynamic Power Management:

There is a new set of host affinity rules for DRS, that can give VMs affinity (or anti-affinity) for a set of hosts in a cluster. This helps a lot when someone is faced with host-based licensing restrictions, for instance. These rules can be set to preferred or required, and both DRS and HA will obey these rules.

DPM now has a set of scheduled tasks to help control it, turning it on and off at certain times of the day if you’d like. Disabling DPM will bring all the hosts out of standby, to help guarantee that no hosts get stuck in a useless state.

vCenter:

The changes to vCenter are really too numerous to list here. The biggest change is that it is 64-bit only, which will be a giant problem for a lot of people who are running 32-bit Windows. There are a lot of other incremental improvements, including a lot more information being presented to the users, often in conjunction with the new features throughout vSphere 4.1.

Whew. Get testing, people!

Why No-Reply Email Is A Bad Idea

Bob Plankers — Thu, 08 Jul 2010 21:17:32 +0000

I absolutely hate no-reply email. I understand why it exists (autoresponders and bounces), but to send an email with no way to respond at all using the same communications medium is ridiculous.

A good example of this is the customer satisfaction survey Red Hat just sent me. It is from a no-reply email address and there is no other email address listed. There is just some text and a URL, and clicking on the URL gets me:

$ dig rhapps.redhat.com ns1.redhat.com
[...snip...]
;; QUESTION SECTION:
;rhapps.redhat.com.             IN      A

A records are overrated.

I generally am a nice guy and let vendors know something is messed up, but there are limits, especially when I’m already on the fence about a negative experience. I’m not going to open a support case with them, because it’ll never get escalated correctly[0]. And there’s no email address to send a quick note to. So it goes unfixed, Red Hat gets added to my mental list of vendors that don’t get it, and I blog about it, which may be worse than a negative survey response.

And for heaven’s sake, if you send out a customer satisfaction survey make sure it works.[1]

——————–

[0] Here’s a test for your organization: can a customer open a support problem against your web site? Will it go to the right place, i.e. the webmasters or someone intelligent who can get things fixed? If not, why not?

[1] This also may mean you should add an external survey service to what you monitor.

Changing VMware vCenter Logging on Windows Server 2008

Bob Plankers — Thu, 08 Jul 2010 18:40:44 +0000

I’ve had occasion, over the last 24 hours, to change the logging level of my vCenter server. It’s been a little more crashy than usual, and I’ve discovered that a number of KB articles are not updated for vSphere 4.0 Update 2 or Windows Server 2008.

If you want to change the log retention policies, you can follow most of the instructions in KB 1004795 but there are two things I think you should know if you’re running on Windows 2008:

1. Your vpxd.cfg file and your logs are actually in:

C:\ProgramData\VMware\VMware VirtualCenter

2. The XML stanza for the logging is commented out in 4.0 Update 2, so remove the lines to make it work. For example, make:

...
  CpuFeatures
 
 
 
    
...

into:

...
   CpuFeatures
 
 
   100
   true
   info
 
 
    
...

You may also want to change the compressOnRoll to ‘true’ as I did above if you are concerned about disk space usage.

This may seem obvious to a lot of people, but for those that only occasionally make changes to vpxd.cfg it helps to have been pre-warned about the differences. At least you’ll vaguely remember that something’s changed… :)

The Lone Sysadmin » Featured

VMworld 2010: Hands-On Labs

Related Posts

VMworld 2010: Saturday

Related Posts

Rebel Alliances

Related Posts

Three Organizational Decisions That Help Me Virtualize

Related Posts

Youth

Related Posts

Get Away to VMworld Contest

Related Posts

Oregon Trail: The Movie

Related Posts

Remembering Swap

Related Posts

Happy System Administrator Appreciation Day!

Related Posts

Rate-Limiting Steps

Related Posts

Per-VM Licensing

Related Posts

Gestalt IT Tech Field Day Seattle

Related Posts

VMware vSphere 4.1: What’s New

Related Posts

Why No-Reply Email Is A Bad Idea

Related Posts

Changing VMware vCenter Logging on Windows Server 2008

Related Posts