Sure, sure, having a standard benchmark to measure virtual machine performance is useful. Customers will swoon over hardware vendors’ published results. Virtualization companies will complain that the benchmark is unfair. Then they’ll all get silent, start rigging the tests, scrape and cheat and skew the numbers so that their machines look the greatest, their hypervisor is the fastest. Along the way it’ll stop being about sheer performance and become performance per dollar. Then CapEx vs. OpEx. Watt per tile. Heat per VM. Who knows, except everybody will be the best at something, according to their own marketing department.
Welcome to benchmarking.
It doesn’t make a damn bit of difference to me, though. I’ll never run VMmark. I’ll never pony up $1200 for a copy of SPECweb2005 or $500 for SPECjbb2005. I’ll never buy four copies of Windows Server 2003 per tile, $999 each, or buy a client PC for every tile. This list alone is thousands of dollars, which I will never pay. Never mind the time it would take to set this whole mess up. God help the vendor who wants to prove they have the fastest machine for virtualization.
It’d be nice if VMware, when they said that “organizations can compare performance and scalability of different virtualization platforms, make appropriate hardware choices and monitor virtual machine performance on an ongoing basis” meant an organization smaller than Fortune 10. It’d be nice to have a benchmark an average guy like me could run. Something I could use to test my own performance tuning, running it before and after each tweak to see if my changes make a difference. Something I could use as a diagnostic, or as a test procedure for a change. Did that last patch mess anything up? Why is my environment really slow? Why am I not getting the advertised performance from these new machines? Something like VMmark Lite, free & open source, might be pretty darn cool.
Instead, I’ll be in meetings explaining to folks why we are maxed out at 30 VMs per server when the vendor says they’ll run 50. Or why we chose VMware over Xen, when Xen claims 100 on the same hardware. I’ll have to remember the line from the FAQ that says “that VMmark is neither a capacity planning tool nor a sizing tool.”
Which begs the question: if it isn’t for use in sizing or capacity planning, exactly what is it good for?
Thats one expensive way to prove very very little.
Here’s the real question… do you have an alternative or are you just complaining?
fruopzexb wzikpa blipx uvncgilm ihyjsguqr kbvzyna fmikjz
I think you’re just complaining. VMmark is not meant to be a sizing or perform capacity planning, but neither is are other benchmark tools. They are to compare hardware. You’re free to use VMmark to compare your own hardware. I guess you should read the FAQ and the WPs again. An alternative to VMmark is a stress testing tool called AutoIT. AutoIT is at http://www.autoitscript.com/autoit