One Workload Per Virtual Machine

This is post #2 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.”

Back when the computing world was built on bare metal we often tried to squeeze more than one workload on a physical machine. We did this to save money. Servers are expensive so we wanted to make the most of each one. What we didn’t account for was how complicated things could get. Applications fought with each other over system libraries and DLLs. Security was complicated. And, most relevant to this series, performance tuning became this multivariate dance involving resource limits and other arcane system witchcraft. In the end we saved a little money on hardware and spent a ton of money on staff time to manage the complexity.

There is a simple fix in this era of hypervisors: run only one workload per virtual machine. More specifically, try to only run one type of work per virtual machine.

Often we think of a workload as a whole system, like a payroll system or this blog. But this blog is really two discrete workloads and two discrete types of work: a multithreaded web server running PHP code and a largely single-threaded MySQL database. On a single system, if we tune the web server to allow more connections we are stealing resources from the database. If we tune the database to have larger caches that uses memory that the web server cannot then use. If we add memory to the system how do we make sure that it goes to the database and isn’t consumed by the web server?

Operating system resource limits are complicated. Hypervisors have great interfaces for resource limits, though, beginning with the virtual machine configuration itself. So why not split the web server and database server up? Then you can tune the web server to consume its whole virtual machine, and you can tune the database to consume its whole virtual machine. You don’t need complicated, time-sucking tools like resource limits to keep each application segregated, because the hypervisor does that for you. You will also get better monitoring visibility into each workload, too. Instead of having one graph that’s the sum of both the web server and database server activity you’ll have two, and will clearly be able to identify problems. You also don’t need to compromise on OS-level tunables. The ideal OS settings for one type of workload are usually not the same as another, so on a shared machine you end up choosing values that are either suboptimal for both or very wasteful.

But what about cost?

Sure, having more virtual machines may add some incremental costs. In the public cloud each instance costs money. You might also need additional operating system licenses. Many monitoring systems are licensed and billed based on the number of checks, so there also might be a rise there because you’ll be checking the health of two OSes. These are hard costs, CapEx, but what we’re really trying to save is OpEx and the soft costs of staff time. Complexity drives up operational costs, and by reducing the complexity of your systems you save more in staff time than you spend on licenses and whatnot. Doubly so if you are using automation tools like Chef or Puppet, where you can set up a configuration for an ideal web server, or ideal database server, once and stamp it out many more times for new deployments and testing.

Truthfully, for very small setups it might not be worth it. For instance, this blog doesn’t get enough traffic to warrant the cost of two separate virtual machines. But many organizations have multi-tier applications that would benefit from separation to reduce complexity and improve performance. If you find yourself using OS-level resource limits at all, spending time determining the root cause of performance issues, or your documentation has a lot of curvy lines looping back to the same few machines I have five words for you: one workload per virtual machine.

——

But what about cost?

1 thought on “One Workload Per Virtual Machine”