The Lone Sysadmin

This morning John Troyer coordinated a bunch of bloggers for a session over at the VMworld 2010 Hands-On Lab facilities in Moscone West. Adam Zimman, Dan Anderson, and Curtis Pope took turns explaining and demoing the lab to us. The lab itself was built as a cloud-oriented system, using software-on-demand and service-on-demand principles, and relying heavily on remotely-hosted equipment in data centers in Miami, FL (Terremark) and Ashburn, VA (Verizon).

The Lab team is really building on what they’ve learned from other years. There are many more labs this year than last, and they’re all self-paced, though there are options for instructor interaction as well if you have questions or want more one-on-one guidance. Self-paced labs means they can do almost unlimited content, and it’s easier to get lots of people through the labs. Last year they had, all totalled, about 7000 lab seat hours. This year they have almost 20,000, with 480 View stations in eight rooms. Dan Anderson, the lab’s lead architect, had some proud things to say about what they’ve done. “The content is killer, the best content I’ve seen yet. If someone sits for four days, eight hours a day, they might be able to get through all of them. But nobody can complain about not having enough stick time,” said Dan.

Perhaps he’s never met some of the curmudgeonly people that attend VMworld. :) But I really appreciate the iterative approach they’ve taken this year to making the labs better. For instance, they learned that pre-registration for the labs didn’t work very well in other years, so it’s all first-come, first-served (FIFO). There’s a check-in station that works with your badge number, and a waiting room with couches and whiteboards and Subject Matter Experts while you wait. The labs will be open from 8 AM until 10 PM every day, too, and they will be offering a prize to “dedicated individuals” (they thought speed and quantity might be the factors, but it isn’t set in stone). They did say the prizes would be something like a pass to VMworld 2011, though, which is very cool.

The hardware and software powering the lab is pretty amazing, with a number of sponsors contributing staff, equipment, and software to make it run. Sometimes on very short notice, too. And in some cases this lab is the largest deployment yet of these technologies. They’re pre-populating lab environments with instances of each lab setup, to avoid the on-demand 5 to 7 minute wait from last year, which is great. They’re worried that they’ll have the prepopulation levels off a little on the first day, but even if you do get caught waiting you can still read the manuals. They estimate that the labs are using roughly 36 TB of RAM (yes, TB) and there’s about 200 TB of storage, between EMC and NetApp, in each data center powering the labs, all connected via NFS. The storage itself is everything from enterprise flash (EFD) to SATA, with the EFD often being used as FastCache to front-end the slower storage.

The stations themselves are Wyse thin clients, with dual monitors and even dual chairs, even though it’s geared for one-on-one learning. It’s all about flexibility and options, which extends to the content itself — the vSphere Sandbox lab is just a deployment of all of their products, for freeform messing around. The Lab team even has redundant wiring to the lab stations, just in case they need it (“We even have redundant chairs!” said Adam). They’re flexible, they’re ready, and they’re hoping that they can set records for the number of happy people in the labs this year. And if you’re not happy, there’s 150 staff floating around to help you out, as well as two lab captains per room.

I’m looking forward to it — labs have always been a highlight of VMworld for me, and these guys are making it even better. I know it’s a lot of work to build, in two months, what usually would be done in a year or two (and then, as Dan said, “throw it on a truck.”). On behalf of all of us, thank you Adam, Dan, and Curtis (and all the others that we didn’t meet). I hope all your hard work is a giant success!

A quick walk past Moscone on Saturday yielded a real life “401: Not Authorized,” most likely because it’s “501: Not Implemented” yet. So we continued on down to AT&T Park to watch the Giants lose to the Diamondbacks, 11-3 (it was 6-1 after the first inning, Barry Zito was having a real bad night). At least we got Joe DiMaggio bobbleheads. Not much else to do, it looks like a bunch of folks led by the inimitable Veeam guys went to the Chieftain. My coworker Steve and I opted for an early turn-in, to hedge against the upcoming late ones. Below are some photos.

On tap for Sunday: a sneak preview of the Labs this morning, brunch with a bunch of the Communities guys, registration and speaker setup at 2 PM, and then nothing. The vmunderground.com WUPaaS is tonight, but I missed the invites, so I might try sneaking in. Otherwise I’m sure other things will be going on. I do need to work on my presentation a bit, and perhaps a trip over into the Sunset district for some excellent Vietnamese at Pho Phu Quoc is in order, followed by a walk down Irving to the beach and the N Judah home.

I hope you’re all traveling well. Stay safe, and remember that if you’ve got a question or are looking for something to do #vmworld is a good start on Twitter. You can also message me, @plankers.

If you were thinking about going to VMworld but can’t make it due to finances Gestalt IT is running a contest where the winner gets airfare, hotel, and a conference pass. This is a very compelling contest, and to win:

Entrants must explain how they plan to “pay it forward” if they get to go to VMworld. Will you start a blog? Write some tutorials? Contribute to a forum or online community? Present to your local VMUG? Get creative and spread the wealth of knowledge you get from the event!

Our panel of judges is made up of none other than the most-excellent roster of past Tech Field Day delegates! They’ve proven themselves to be independent-minded and knowledgeable, and we’re sure that they will pick the best entry!

I’ll be one of the judges — I’m looking forward to seeing some really great proposals! Go over to Gestalt IT for all the details.

Public voting is open now for VMworld 2010 sessions. I’ve already gone and voted for my own session, “Tuning Linux for Virtual Machines – TA8102,” in the Technology & Architecture track, as well as a number of other heavyweight presentations by Duncan Epping, Scott Drummonds, Mike Laverick, David Davis, Steve Chambers, Chad Sakac, Edward Haletky… I’m missing a lot of people in my list here, but the point is that there are tons of presentations by people who really know what they heck they’re talking about.

If you are thinking about going to VMworld this year you should go and pick the content YOU would like to see in the conference. It’s pretty rare that attendees get to help set up a conference in this way, so don’t squander it. You only have a week! Go!

If you’re on the fence about going, or don’t think that your company will send you, now is the time to start working on that. VMworld 2010 is the premier VMware conference and a great training opportunity on nearly every VMware-related topic available. You’d pay the same amount for a training class on a single topic. Plus, San Francisco is beautiful right around Labor Day…

Vaughn Stewart posted this morning that the VMworld Call For Papers closes on Friday, April 9. Last Friday, I posted that it closed, well, last Friday. I think it’s safe to say that both of us encourage you to get moving!

You don’t need slides prepared yet, just a 250 word or less session description, three takeaway points you want to convey, a short biography for yourself, a short description of your speaking history, and knowledge of your own mailing address. :) Took me 30 minutes to do.

As for the date, I guess it depends on where you look! The main VMworld page says there are two days until things close. The “Key Dates” page says April 2:

I’m just glad I’m not going insane… could have sworn it was April 2! I can still edit my submission, so I guess things are still open. While the extension is great news for procrastinators, I hope that the miscommunication over dates can be resolved. Regardless, you have 30 minutes between now and Friday, get your submission in before something else changes! :)

Heck yeah, I was approved to go to VMworld 2009 today.

In case you’re still trying to convince someone you should go, here’s the approach that has worked for me for the last three years, amidst constant budget problems and skepticism about virtual environments:

1. VMworld is the best opportunity & best value for VMware software training in the United States during the year. There are hundreds of sessions, labs, and other hands-on opportunities. You’ll learn better ways to do what you’re already doing, good ways to start doing things you want to do (like implementing vSphere 4, SRM, Lab Manager, View, etc.), and be able to talk to people who are already doing it to find out how well it really works. It’s also a great place to see what the future directions are for virtualization.

2. On top of the conference sessions itself the vendor showcase is very useful. See new product releases from vendors, learn about options for storage, networking, hardware and software to make your virtual environments even better.

3. Virtualization is the way that all future IT operations are going to be run. As such, a trip like this is an investment in your company’s IT future. Additionally, compared to the cost of hiring a consultant, a week of VMworld can be quite inexpensive. It also keeps the knowledge in-house. Not saying consultants aren’t handy, because they can be very knowledgeable and useful, but I’m always a fan of the local IT guys being the long-term experts on their own systems. VMworld helps with that.

4. The airfare is quite inexpensive from most places in the U.S., and with San Francisco there isn’t the perception of some lavish trip to exotic places to play golf. If you’re coming from outside the country the exchange rate is probably still quite favorable, too. And if your significant other wants to go, but doesn’t want to get stuck with a bunch of dorks talking about Extended VMotion Compatibility and HBA queue depths, Scott Lowe’s wife Crystal is organizing some touring, too. Plus, it’s the long Labor Day weekend, it’ll be awesome!

5. Networking with VMware staff, VMTN & Communities celebrities, and all those vExpert rock stars is handy. The virtualization community, while it’s growing rapidly, is still pretty small, and you can meet and buy beer on your expense account for all those folks whose blog and forum posts have saved your duff all year long.

Okay, #5 probably isn’t super useful in your justifications. But I do owe a lot of people beer. And there’s lots of great beer in San Francisco… Now to just find an expense account…

So get it together, people! Get registered, get your flights, and get moving!

It’s budget season where I work. We operate on a chargeback model, so this means that I get to guesstimate what I’m going to need to spend next year, look into my crystal ball to see what I’m going to bring in, and then figure out if I can keep my chargeback rates the same or not. Infrastructure folks like me are at the bottom of the budgetary food chain, so changes to my rates have serious implications for people higher up. It’s been pretty nice to be able to drop the chargeback rates of virtual machines year after year, as hardware gets faster but the workloads remain fairly constant. Dropping your prices is definitely easier than raising them.

Something new in my budget this year is explicit funding for several people to go to VMworld. Over the last few years I’ve been able to attend as part of my organization’s training budget. With budgets shrinking I want to ensure that some folks get to go in 2009, so I’m building it into my service fees.

If you aren’t in charge of your budget you should find the person who is and lobby them now to put funding in for VMworld 2009. There are lots of good reasons they should send you.

• It’s a training opportunity, with hands-on labs where you get to use the software, learning best practices and good ways of solving problems.
• It’s an opportunity to talk to other people having the same problems and challenges as you, finding out what works and what doesn’t work before you spend a lot of time on it.
• It’s an opportunity to talk to VMware staff who can answer your questions directly, like by asking questions in sessions.
• It’s a way to see new technology, so you can be ahead of upgrade curves and be able to decide whether a particular technology, like SRM or VDI, is useful to you without starting a big in-house evaluation.
• VMworld is a great place to see all the other hardware and software vendors and the new technologies they are rolling out.
• It’s a way to get access to all the VMworld content, so you can look it up when you get home and implement all the good ideas that are presented in the various sessions, including the ones you missed because there were too many good sessions scheduled at the same time. :-)

So get going! I don’t want to hear next fall that you can’t attend. If you want to go plant the seed now in your management’s heads.

Dr. Stephen Herrod is the featured speaker of the VMworld 2008 day 2 general session. He’s the CTO & senior VP of R&D. This post will be more notes than a coherent piece. I’ll follow up with some thoughts later today.

• Goal of the infrastructure layer is to aggregate resources and be as efficient as possible.
• vCompute layer: things like FlexMigration. Focused on the VM, making it as powerful as possible. Grown from 2 vCPUs to a future 8 vCPUs, 40 Gbps, 256 GB per VM, 200000+ IOPS.
• Next generation of resource pools, up to 64 nodes in a single cluster, 4096 processor cores, 64 TB of RAM, 6 million IOPS, all running under DRS.
• Distributed Power Management continuing in development. Their VMmark exercise saved 50% of the power with DPM enabled.
• vStorage: leverage all the advanced technology of the storage in a common framework. VMFS continues forward, with enhancements to Storage VMotion. Thin provisioning will be added, and linked clones will enable better patching methods. Focus will also be on vendor management tools, enabling an API for them to interact with the VDC-OS.
• Thin provisioning is great, but also a lot of work to help with overcommittment problems. They also want to leverage hardware methods of lazy allocations, too.
• vNetwork: VMotion doesn’t preserve state between ESX servers, which is a problem for things like security. New model is the vNetwork Distributed Switch, which is a cluster-wide implementation of the virtual switch, and VMs will take their network profiles with them when they VMotion. First third-party switch is the Cisco Nexus 1000V, which is nice because network admins can interact with the virtual switches like any other Cisco equipment.
• Application vServices layer: everything will work for existing application loads, as well as new ones that we don’t know about yet.
• vApp: existing VMs can be wrapped in the vApp container. Based on OVF, encapsulates all the different VMs that make up an application, also represents SLA information which represents a contract with your VDC-OS.
• A variety of technologies to help with planned and unplanned downtime, like NIC & HBA teaming, VCB, HA, VMotion, Storage VMotion, and at a high level, Site Recovery Manager.
• Fault Tolerance: focused on unplanned downtime for an application. Idea is that a shadow copy of a VM is running somewhere else, so if something happens the other VM will be available. A demo of the FT technology by Mark Vaughn of First American Corp. FT is simple to enable with a right-click in the new vCenter client. It basically starts a VMotion to copy the running state over, and then enables vLockstep to mirror the activities of the primary VM. FT helps with a lot of human error, cords coming out, powering off the wrong box. When something happens and FT kicks in it will automatically start reprotecting itself within the same resource pool.
• Security vServices: be able to describe the services in the vApp definition. The building block for the security services is a set of APIs called VMSafe. Security software is working outside the VMs themselves. Relies heavily on the distributed network switch. This means new VMs are automatically protected. It’s “right-sized” security capacity, which grows easily with your environment.
• Management vServices: VirtualCenter becomes vCenter, and way more extensible. Partners can plug in for extensibility and also for “pane of glass” sorts of things. ConfigControl, Orchestrator, CapacityIQ, and Chargeback all new modules to do cool things with the infrastructure.
• AppSpeed helps monitor applications and work with the VDC-OS to take actions. It discovers the topology of services in a non-intrusive way. Then looks at QoS from an end-user perspective, and can deduce when things are wrong. It’s a totally new way to collect information in the virtual framework. A demo from Asaf Wexler, formerly of Beehive (now with VMware). As the application is used a map of the dependencies is created automatically. The demo vCenter plugin was built overnight by the AppSpeed team. Stats available can help identify root causes of problems. Policy engine can take actions based on stats gathered, to fix problems automatically.
• vCenter: lots of work on alarm capabilities. Getting ported to Linux, also to be shipped as a virtual appliance. Adding multi-platform client technology to run on Macs, Linux, iPhones, etc. On the iPhone, if you shake it you trigger a disaster recovery scenario (joke!). :-)
• Cloud vServices: “cloud” most abused phrase since “virtualization.” A model to quickly use resources, to pay for only what you need, accessible with standard protocols, scalable & elastic, and economies of scale. “Local cloud” exists in your own data center. Needs to be more flexible. App compatibility maintained into the cloud, without having to rewrite an app. Standardization removes complexity in apps and lowers switching costs. Last, provides multi-tenancy in a resource pool, where others can come in and be assured their security and performance needs are met.
• vCloud APIs: image management, user accounts, chargeback, mobility, all APIs needed to bridge the gap between local clouds and off-premise clouds.  Clouds are federated, and the vApp/OVF definitions help determine where an app could run, whether it can be outside a firewall, for example, or it needs to stay internal.
• VMware View: targets applications and data, following a user and not their hardware. Use mobile devices, home computers, etc. securely and with proper management. View evolves VDI, User experience changes based on where you are, whether on a WAN, LAN (richer experience), or local (with graphics cards, etc.).  New technologies to cache VMs locally. Unity in Fusion and ACE contribute to these new technologies. A very thin layer to avoid other OS software. Demo by Jerry Chen of VMware, demonstrating setup tasks using View Manager and View Composer. Linked clones saves a lot of disk space while speeding the provisioning time. New client hypervisor runs VMs locally on hardware but lets you manage the VM from View Manager. Demo of adding Google Chrome to all the desktops using ThinApp, update master image, then notifies clients to restart. Jerry Chen then rickrolled the attendees to prove that Chrome works on the clients. Thanks. :-)

In the end, irtualization is proving to be a broad problem-solver. :-)