Bob Plankers

Bob Plankers is a man who wears many hats. In his professional career he has spent three decades leading teams that design, build, operate, and talk about reliable, secure, and compliant IT infrastructures, both in the public sector and in the high tech industry. He currently focuses on applied security, compliance, and operational resilience for virtualization and cloud platforms.

Store Logs Centrally

by

This is post #7 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” It is universally true that over time a thin-provisioned virtual machine will slowly expand on disk. This happens because of the way files are deleted from disks. When files are written to disk the storage subsystem writes those blocks out, and the thinly-provisioned disk file grows a bit. But when a filesystem deletes a file all it does is update its own internal storage map to “forget” about the file. Nothing ever removes the data from the disk itself (which is why undelete utilities work, and why there are stringent procedures for securely erasing a …

Read More

Use elevator=noop For Linux Virtual Machines

by

This is post #6 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Modern operating systems are fairly modular, and often have different modules to deal with memory, network I/O, and CPU scheduling. Disk I/O is no exception under Linux, as there are usually four different schedulers a sysadmin can choose from. Red Hat had a nice write-up on these a few years back and it remains relevant today: The Completely Fair Queuing (CFQ) scheduler is the default algorithm in Red Hat Enterprise Linux 4. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O …

Read More

Don’t Start GUIs and X Windows On Servers

by

This is post #5 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” There are stories out there about system administrators with intermittent performance problems on their Windows servers. Like many problems, it would never happen when the administrator was around, but when they walked away from their servers suddenly the issues would crop up. It was the OpenGL screensaver, sucking up all the CPU. GUIs make life easier sometimes, especially if you aren’t as familiar with an environment as you’d like to be. However, they come at a cost: additional RAM, CPU, and disk overhead. In my travels I’ve seen virtual environments more than double their consolidation …

Read More

The Best OS Installation Is Minimal

by

This is post #4 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” In this day & age of virtualization and clouds the best choice for an OS installation, be it a virtual machine template or a physical machine, is a minimal install. From there you can use a configuration management tool like Chef or Puppet to add exactly the packages you need for the host to perform its functions. Doing a minimal install has several advantages: A minimal installation doesn’t install a lot of content that will just sit there and consume disk space. As a result your template VMs will be really small, which leads to fast provisioning …

Read More

Minimal Kickstart File for Red Hat Enterprise Linux 6, CentOS 6, Oracle Linux 6 Virtual Machines

by

This is a helper post in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Here’s a minimal kickstart file for a virtual machine installation of RHEL 6, CentOS 6, or Oracle Linux 6. I use this in conjunction with tools like Puppet and Chef to do the rest of my system configuration. I provision 30 GB thin-provisioned disks to each VM, leaving the rest of it unallocated until needed. While this leads to a disk overcommitment situation that’s manageable, and a tradeoff to ensure more standardization and ease of administration & automation. I remove a number of packages that we don’t necessarily need on a virtual machine, either …

Read More

Performance Tuning and Configuration Management Go Hand In Hand

by

This is post #3 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” I’ve really become a fan of configuration management tools like Chef and Puppet. Those types of tools help keep my systems in sync, help enforce standards I set for my systems, and help me rapidly deploy new systems that look exactly like my existing systems. When I’m doing performance testing & tuning these aspects are helpful. It’s easy to deploy a new virtual machine that looks exactly like the production VMs you have. Tuning is all about making a lot of little changes and seeing what they do, so by building test systems from the …

Read More

One Workload Per Virtual Machine

by

This is post #2 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Back when the computing world was built on bare metal we often tried to squeeze more than one workload on a physical machine. We did this to save money. Servers are expensive so we wanted to make the most of each one. What we didn’t account for was how complicated things could get. Applications fought with each other over system libraries and DLLs. Security was complicated. And, most relevant to this series, performance tuning became this multivariate dance involving resource limits and other arcane system witchcraft. In the end we saved a little money on …

Read More

For The Best Performance Use a Recent Operating System and a Recent Hypervisor

by

This is post #1 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Just like in construction, performance tuning is most effective when you have a good foundation. If the underpinnings of your efforts are weak you won’t be able to build a skyscraper, just a small office building. While that’s still better than nothing, there is often a lot to be gained by using the newest versions of your OS & hypervisor of choice. Some quick examples: Windows Server 2012 R2 added Virtual Receive-side Scaling, which allows the load from network traffic to be processed by multiple virtual CPUs. VMware vSphere 5.5 added support for latency-sensitive VMs, …

Read More

Linux VM Performance Tuning, Take 3

by

A couple years ago I started rewriting my venerable Linux VM Tuning Guide, as a series of posts throughout the month of December inspired by Sysadvent (which is running again, and always has good stuff). I called it “Tuningmas” like Christmas. Yeah, corny, I know, but hey. I got about three posts out, then some life stuff happened, and the rest of my draft posts never made it out. 2011 was a year of tremendous highs and tremendous lows. 2012 was a blur, and 2013 has been, uh, sporty, too, but things are settling down. So I’m trying again. I’m not much of a revisionist blogger but the new series will deprecate all my old work, so I’ll likely go …

Read More

Updates to My vCSA 5.1 to 5.5 Notes

by

I just updated my notes on upgrading the VMware vCenter Server Appliance from 5.1 to 5.5. I added a couple of things that have become issues: Remove all non-standard users from SSO before the upgrade. If you added users to the 5.1 Single Sign-On system directly those users will be copied to the 5.5 vCSA as members of SYSTEM-DOMAIN. Unfortunately they will then become trapped, undeleteable & unchangeable, as VMware didn’t think to make the SYSTEM-DOMAIN an editable domain. You can see them, and you can still log in, but you cannot remove them or change their passwords. Your only recourse is to remove the permissions for that user from vCenter, which still means they can log in, but won’t have …

Read More

+ More Posts +