Performance Tuning and Configuration Management Go Hand In Hand

This is post #3 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.”

I’ve really become a fan of configuration management tools like Chef and Puppet. Those types of tools help keep my systems in sync, help enforce standards I set for my systems, and help me rapidly deploy new systems that look exactly like my existing systems. When I’m doing performance testing & tuning these aspects are helpful.

1. It’s easy to deploy a new virtual machine that looks exactly like the production VMs you have. Tuning is all about making a lot of little changes and seeing what they do, so by building test systems from the same “recipe” that you used to build the production systems you greatly increase the odds of success. These tools also turn provisioning into a rapid process. I can deploy a test VM that’s identical to my production VM in under 2 minutes, with zero interaction from me once the process starts.
2. Puppet and Chef can reset a system to your default configuration. When I test things I disable the Puppet agent, change a parameter, hit the system with a load generator and measure the performance, then run the Puppet agent manually to put things back the way I expect before moving on.
3. When you find a better configuration you can guarantee that all your systems get that new configuration by coding the changes into the manifests or recipes. That means two things. First, you won’t update 10 of your 12 web servers with a new configuration, go to lunch or take a phone call, and forget the last two. This is really, really common. Humans are terrible at context switching and we forget things when we’re interrupted. We also think we’re rock stars and don’t do this, but in reality we just forgot that we left something undone. Second, you won’t have to remember all the things you changed. Your notes are now code that Puppet uses to build the new systems, accessible by your whole team. And if you’re good you’ve got those in version control, so you can see exactly who changed the web server configuration last week and why.

These are just the reasons that these powerful tools and ideas are useful for tuning. Once you get into it you’ll see many more ways they’re helpful.

This looks complicated. I don’t have time for this.

You do have time, because here’s the deal: these methods save so much staff time and improve process so much it’s gross negligence not to use them. Which means that when your company’s management figures this out they’ll do one of two things. They’ll either give you the go-ahead to implement it yourself based on a nice proposal you’re going to write for a pilot project, or they’ll give the go-ahead to the consultant that clued them in. Make it your great idea, and make them give you some time to do it away from your normal duties. This is an inflection point in IT, choose to be on the right side of it.

It took me about 60 minutes to get a CentOS 6 VM running as a Puppetmaster, following their installation documentation and using their yum repositories. To start out I’d suggest just using the built-in web server it ships with, which doesn’t scale well past about 10 nodes but will get you moving in the right direction. By the time you run into a problem here you’ll know how to fix it. You can also download the Puppet Enterprise VM which uses the drug dealer model – first 10 nodes are free – but it has fancy add-ons all preconfigured. It’s a lot to bite off at once. Go simple and build your own.

Second, I built a simple Subversion repository to check my manifests into. I chose Subversion because I’m familiar with it, but you might also check out git. Or go talk to a developer in your company, and use what they’re using. Why use version control? For starters, this is a best practice for software developers, and what you’re going to be writing is code. You can also use Geppetto, a Puppet development environment, to interact with svn or git. Download Geppetto, install it, and practice checking things in and out. When you’re ready, check all of /etc/puppet into it from your Puppetmaster, and check it back out using Geppetto so you can start coding. When you code, check the changes back in to the repository, then check them back out on the Puppetmaster and restart it.

Third, start small. Create a couple of test VMs, and use Puppet to manage /etc/motd. Then manage everything in /etc/sshd along with the services. Manage a user account. Go through the tutorials. Download the Puppet Labs NTP module from Puppet Forge, check it into the repository, and use it to configure NTP on your hosts. It’s a great first Forge module. Keep working until your test VM has all the customizations you normally make. Some things will be harder than others. SELinux and IPv6 management leave a lot to be desired, for instance.

Last, retrofitting existing systems is tough. If you need to do it start small, like /etc/motd or a user account. Wait for the next version of Red Hat Enterprise Linux, or a new Ubuntu, or a need for a new template VM, but not too long. Just a good clean starting point to go whole-hog with Puppet.

I’m a Linux guy, and this is a Linux post, and Puppet and Chef are really UNIX tools despite their claims to the contrary. If you’re managing Windows you might check out what you can do with SCCM and Group Policies, too.

——

Image of the KUKA robot © 2003  KUKA Roboter GmbH, licensed into the public domain, provided via the Wikimedia Commons.