Best Practices

Do Not Collect System Performance Data From Guest OSes

by

This is post #12 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Fans of the 12th Doctor Who have often heard the phrase “the Doctor lies.” The explanation for his lies is that, because he skips around in time, he knows things that others cannot know yet. Hypervisors are like that, too. Guest OSes don’t know that they aren’t the only OS on the hardware, and the hypervisor lies to them about things like CPUs, RAM, and things like system timers because, like the Doctor, the hypervisor is skipping a VM forward in time. And that’s the rub – only the hypervisor knows what the truth is. …

Read More

Adjust vm.swappiness to Avoid Unneeded Disk I/O

by

This is post #11 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” The Linux kernel has quite a number of tunable options in it. One of those is vm.swappiness, a parameter that helps guide the kernel in making decisions about memory. “vm” in this case means “virtual memory,” which doesn’t mean memory allocated by a hypervisor but refers to the addressing scheme the Linux kernel uses to handle memory. Even on a physical host you have “virtual memory” within the OS. Memory on a Linux box is used for a number of different things. One way it is used is internally for buffers for things like network …

Read More

Zero Out Free Space

by

This is post #10 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” When we talked about the rationale behind storing logs centrally one big reason was thin-provisioned virtual disks. Those disks grow over time because filesystems on a virtual machine currently have no way to tell the underlying storage that they’re done using certain blocks on disk. There is a way to make these VMs thin again, and I wrote about it as step 9 in my guide to preparing Linux Template VMs. In short, we run a script on the VM that writes zeroes to most of the free space on the VM: #!/bin/sh # Determine …

Read More

Disk Partition Alignment Is Still Important

by

This is post #9 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” I have written about this almost yearly (beginning all the way back in 2006), but even now I routinely run across something, like a virtual appliance, that has poor partition alignment. What’s the big deal? In short, misaligned I/O is killing your disk performance. Blame Logical Block Addressing, or LBA. Back in the day, a BIOS interacted with drives by knowing the exact geometry of the drive, namely how many cylinders, heads, and sectors were on a disk (CHS). Unfortunately that limited the size of the drives that could be used, and ignored some basic …

Read More

Gain 30% Linux Disk Performance with noatime, nodiratime, and relatime

by

This is post #8 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” According to Red Hat’s Enterprise Linux 6 Power Management guide, the POSIX family of standards for operating systems “requires that operating systems maintain file system metadata that records when each file was last accessed.” This is called “atime” (a is for access) and is one of three timestamps Linux filesystems keep for their files (the other two are mtime and ctime, modify and change times). You can see these with the ‘stat’ command: $ stat /bin/ls File: `/bin/ls’ Size: 109208 Blocks: 216 IO Block: 4096 regular file Device: fd00h/64768d Inode: 519 Links: 1 Access: (0755/-rwxr-xr-x) …

Read More

Store Logs Centrally

by

This is post #7 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” It is universally true that over time a thin-provisioned virtual machine will slowly expand on disk. This happens because of the way files are deleted from disks. When files are written to disk the storage subsystem writes those blocks out, and the thinly-provisioned disk file grows a bit. But when a filesystem deletes a file all it does is update its own internal storage map to “forget” about the file. Nothing ever removes the data from the disk itself (which is why undelete utilities work, and why there are stringent procedures for securely erasing a …

Read More

Use elevator=noop For Linux Virtual Machines

by

This is post #6 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Modern operating systems are fairly modular, and often have different modules to deal with memory, network I/O, and CPU scheduling. Disk I/O is no exception under Linux, as there are usually four different schedulers a sysadmin can choose from. Red Hat had a nice write-up on these a few years back and it remains relevant today: The Completely Fair Queuing (CFQ) scheduler is the default algorithm in Red Hat Enterprise Linux 4. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O …

Read More

Don’t Start GUIs and X Windows On Servers

by

This is post #5 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” There are stories out there about system administrators with intermittent performance problems on their Windows servers. Like many problems, it would never happen when the administrator was around, but when they walked away from their servers suddenly the issues would crop up. It was the OpenGL screensaver, sucking up all the CPU. GUIs make life easier sometimes, especially if you aren’t as familiar with an environment as you’d like to be. However, they come at a cost: additional RAM, CPU, and disk overhead. In my travels I’ve seen virtual environments more than double their consolidation …

Read More

The Best OS Installation Is Minimal

by

This is post #4 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” In this day & age of virtualization and clouds the best choice for an OS installation, be it a virtual machine template or a physical machine, is a minimal install. From there you can use a configuration management tool like Chef or Puppet to add exactly the packages you need for the host to perform its functions. Doing a minimal install has several advantages: A minimal installation doesn’t install a lot of content that will just sit there and consume disk space. As a result your template VMs will be really small, which leads to fast provisioning …

Read More

Minimal Kickstart File for Red Hat Enterprise Linux 6, CentOS 6, Oracle Linux 6 Virtual Machines

by

This is a helper post in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” Here’s a minimal kickstart file for a virtual machine installation of RHEL 6, CentOS 6, or Oracle Linux 6. I use this in conjunction with tools like Puppet and Chef to do the rest of my system configuration. I provision 30 GB thin-provisioned disks to each VM, leaving the rest of it unallocated until needed. While this leads to a disk overcommitment situation that’s manageable, and a tradeoff to ensure more standardization and ease of administration & automation. I remove a number of packages that we don’t necessarily need on a virtual machine, either …

Read More

+ More Posts +