performance

Use elevator=noop For Linux Virtual Machines

by

This is post #6 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Modern operating systems are fairly modular, and often have different modules to deal with memory, network I/O, and CPU scheduling. Disk I/O is no exception under Linux, as there are usually four different schedulers a sysadmin can choose from. Red Hat had a nice write-up on these a few years back and it remains relevant today: The Completely Fair Queuing (CFQ) scheduler is the default algorithm in Red Hat Enterprise Linux 4. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O …

Read More

Don’t Start GUIs and X Windows On Servers

by

This is post #5 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” There are stories out there about system administrators with intermittent performance problems on their Windows servers. Like many problems, it would never happen when the administrator was around, but when they walked away from their servers suddenly the issues would crop up. It was the OpenGL screensaver, sucking up all the CPU. GUIs make life easier sometimes, especially if you aren’t as familiar with an environment as you’d like to be. However, they come at a cost: additional RAM, CPU, and disk overhead. In my travels I’ve seen virtual environments more than double their consolidation …

Read More

The Best OS Installation Is Minimal

by

This is post #4 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” In this day & age of virtualization and clouds the best choice for an OS installation, be it a virtual machine template or a physical machine, is a minimal install. From there you can use a configuration management tool like Chef or Puppet to add exactly the packages you need for the host to perform its functions. Doing a minimal install has several advantages: A minimal installation doesn’t install a lot of content that will just sit there and consume disk space. As a result your template VMs will be really small, which leads to fast provisioning …

Read More

Minimal Kickstart File for Red Hat Enterprise Linux 6, CentOS 6, Oracle Linux 6 Virtual Machines

by

This is a helper post in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Here’s a minimal kickstart file for a virtual machine installation of RHEL 6, CentOS 6, or Oracle Linux 6. I use this in conjunction with tools like Puppet and Chef to do the rest of my system configuration. I provision 30 GB thin-provisioned disks to each VM, leaving the rest of it unallocated until needed. While this leads to a disk overcommitment situation that’s manageable, and a tradeoff to ensure more standardization and ease of administration & automation. I remove a number of packages that we don’t necessarily need on a virtual machine, either …

Read More

Performance Tuning and Configuration Management Go Hand In Hand

by

This is post #3 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” I’ve really become a fan of configuration management tools like Chef and Puppet. Those types of tools help keep my systems in sync, help enforce standards I set for my systems, and help me rapidly deploy new systems that look exactly like my existing systems. When I’m doing performance testing & tuning these aspects are helpful. It’s easy to deploy a new virtual machine that looks exactly like the production VMs you have. Tuning is all about making a lot of little changes and seeing what they do, so by building test systems from the …

Read More

One Workload Per Virtual Machine

by

This is post #2 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Back when the computing world was built on bare metal we often tried to squeeze more than one workload on a physical machine. We did this to save money. Servers are expensive so we wanted to make the most of each one. What we didn’t account for was how complicated things could get. Applications fought with each other over system libraries and DLLs. Security was complicated. And, most relevant to this series, performance tuning became this multivariate dance involving resource limits and other arcane system witchcraft. In the end we saved a little money on …

Read More

For The Best Performance Use a Recent Operating System and a Recent Hypervisor

by

This is post #1 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Just like in construction, performance tuning is most effective when you have a good foundation. If the underpinnings of your efforts are weak you won’t be able to build a skyscraper, just a small office building. While that’s still better than nothing, there is often a lot to be gained by using the newest versions of your OS & hypervisor of choice. Some quick examples: Windows Server 2012 R2 added Virtual Receive-side Scaling, which allows the load from network traffic to be processed by multiple virtual CPUs. VMware vSphere 5.5 added support for latency-sensitive VMs, …

Read More

Statistics Rollups Are Evil

by

It’s pretty common for statistics-gathering software, like MRTG, Cacti, VMware vCenter, etc. to roll statistics up over time by averaging them. This helps save space, as well as cut down on the processing needed to look at & graph the data. The problem is that the process is lossy. These systems save disk, memory, and CPU by averaging the data over longer and longer time periods. Those averages remove spikes and make the data less and less representative of what actually happened on your system or network. It also makes it damn near useless for planning and troubleshooting. Let’s start with an example I drew up in Excel to simulate something like vCenter recording an application server’s CPU load every …

Read More

5 Minutes with vCenter Operations Manager 5

by

I installed the new vCenter Operations Manager 5 about an hour ago. Whoa. I had the pleasure of meeting Kit Colbert, Praveen Kannan, and Martin Klaus last April when they were in town, and I know they, and the rest of the team, have been doing a lot of work on the product. It really shows. Here are my first impressions, literally. 1. The install procedure is very straightforward, in that I was able to get it fully installed without reading any documentation. The product is a vApp now, and comes with two components, the Analytics VM and the UI VM. You need two IPs, and despite the OVF import process prompting for the IPs those IPs need to be …

Read More

How Large Your Linux Swap Partition Should Be

by

This is post #4 in my December-long series on Linux VM performance tuning, Tuningmas. One of those timeless questions in system administration has always been “how much swap space do I configure on my server?” The old rule used to be twice the amount of memory, but does a server with 256 GB of RAM really need a half terabyte of swap? And what about VMs? Swapping on VMs is a serious performance drag. Would it be a good idea to just disable swap completely? One thing to consider is that there’s a tunable kernel parameter, /proc/sys/vm/swappiness, that controls the tendency of Linux to scavenge inactive memory pages and swap them out. It is a number from 0 to 100, …

Read More

+ More Posts +