OpenStack, Lock-In, Support Costs, and Open Source Free Lunches

Since I posted my missive about OpenStack not being our savior from lock-in or support costs I’ve had a number of comments and discussions about it. The discussions generally start from the point of view that I’m wrong. Let’s take a look at a few of these.

Also, it might seem like I’m picking on Randy Bias and Greg Ferro a little here but Randy seems like a good guy, and Greg is a friend, so there’s no animosity. Just point/counterpoint.

TL;DR version: OpenStack is cool but isn’t some magic tech that prints money, open source doesn’t mean you don’t have to pay someone to support a service built on it, customized open source and custom solutions using open source don’t mean someone can just pick up support for it, Linux and open source is actually scary to the bulk of IT staff out there, and every organization in the world wants less IT. More humans in your IT department isn’t less IT, it’s the worst of more IT.

“You’re being nonsensical and conflating switching costs with lock-in. They are not the same thing.”@randybias, CTO of the OpenStack reseller Cloudscaling.

Dictionary.com says that lock-in is “an act or instance of becoming unalterable, unmovable, or rigid” or “commitment, binding, or restriction.” Seems about right for an organization that is unable to switch from one product to another.

How does lock-in happen in a corporation? One way is that the feature set offered by one solution doesn’t exist anywhere else. If you’re using those features and need them you can’t switch. Another way is that an organization doesn’t have enough money to execute a change. A change in infrastructure costs money in terms of staff time spent switching, license & support fees for both environments, and hardware to operate both environments in parallel for the duration of the switch. Last, sometimes there isn’t an exit strategy. There are quite a number of cloud solutions out there, especially SaaS ones, where a data export just isn’t a feature. For IaaS I always argue that as long as you can boot a VM from an ISO image there will always be an exit strategy.

OpenStack doesn’t free us from any of these things that contribute to lock-in. It doesn’t magically solve the feature set problem. Though it may save me some money in licensing it doesn’t magically put money in my pocket to pay staff to do the work to switch. It doesn’t buy me new hardware. It doesn’t magically require less support, though the form that support comes in may be different from other hypervisors. And it doesn’t have a magic migration path from other hypervisors.

“Free” software is rarely free to implement, and I’m sure Cloudscaling sends bills to people despite their basis in open source software. :) Open source cloud software is just another option to consider, like everything else out there, with its own pros and cons.

“*You* control your destiny with open source solutions and this is why open source is so broadly adopted in the enterprise.”@randybias

“You can just get someone else to support your open-source software deployment.” – a bunch of people on Twitter

First, I call shenanigans on the idea that open source is broadly adopted in the enterprise, at least in the ways we’re talking about here. Might be headed that direction in some specific ways, like Hadoop, but I think this is optimism. Most open source software running in enterprises is there inadvertently, as firmware or an appliance, and not usually as a conscious choice. It’s actually a problem for many people trying to use VMware’s vCloud Director, as those deployments require Linux knowledge, and that’s something that most places don’t seem to have. Lack of Linux knowledge isn’t a plus for OpenStack, seeing as OpenStack is deeply, and in some ways quite arcanely Linux-based.

If there were broad, meaningful adoption of open source infrastructure we’d see more Linux expertise out there. We’d also not have to argue constantly with vendors about providing CLIs. Seriously, who chooses a single pane of glass over a CLI thinking it saves time? Windows admins!

Second, enterprises are risk-averse, and those that aren’t IT hosting companies tend not to deploy anything, much less open-source software, without a vendor standing behind it in some way. Why? It’s a hedge against staff knowledge gaps and personnel issues, at least. The only way a company will ever really control its own destiny with open source is if it forks the code base and hires its own developers to work on the software. That might be feasible for companies like Google where IT is their product, but for most others it’s not even a consideration, and so they will be at the mercy of the open source developers just like they are at the mercy of the commercial vendors. When is the last time you had a say in the way Red Hat Enterprise Linux was built? Does Richard M. Stallman really represent your best interests? Never, and hell no.

Besides, you’d still have to pay the developers you hired to support the forked codebase. Does that cost more or less than an equivalent commercial solution?

Right now it looks like most of the commercial offerings of OpenStack are customized in some way, and Dominic Wellington (@dwellington) said it best: “OSS also has lock-in: custom development on one stack is rarely portable to another. #TANSTAAFL”

Simple experiment: call Piston Cloud and ask them if they’ll support a Cloudscaling environment.

“The cost per host for VMware/Citrix quickly becomes unsustainable… if you are building a cloud platform for 50K to 100K hosts.” – rearranging Greg Ferro’s argument just a little. Greg is a networking consultant in the UK.

Neither of us know what hosting providers pay for VMware or Citrix licensing so we probably shouldn’t talk about whether it is cost-competitive or not. I guess I’m not really talking about hosting providers, though, I’m talking about more traditional enterprises. Hosting providers have their own economics and motivations that are much different from companies that don’t offer IT services as their primary product. In some ways VMware is also a competitor in these spaces. Does Rackspace want to run their competitor’s software? Probably not. Makes total sense for them to develop their own software and maintain a small army of coders to make it work. Even their choice to give it away has benefits, too, in that others will develop the ecosystem for them for free.

“When licensing software for more than 10K units, it’s more likely to be cheaper and better to use humans. Even better, you can design and more tightly specify your solutions to exactly match your business needs.”Greg Ferro

Sounds like the ultimate in lock-in to me! It’s a classic buy-versus-build decision. For a few operations build makes more sense because of the product they’re delivering, and by building their own infrastructure they gain a competitive advantage. For most it makes more sense to buy, because IT services are not their competitive advantage, not the market they are in, and not a market they want to be in. This is the same rationale that led Facebook to release the Open Compute Platform specs — it’s not their core business.

Also, I’ve never correlated more humans with more tightly specifying anything, except maybe a new internal HR system. And what is an HR system? Overhead you wouldn’t need if you had automated instead of hiring. :)

“While it is complex and ‘hard’ to manage a lot of human meat puppets to deliver projects, Amazon has proven that it’s possible. And Facebook. And Google. Your business can also achieve these outcomes if you are smart and able to refocus on your IT resources.”Greg Ferro

Apples and oranges. This is confusing the companies where IT services are their product with companies that consume IT resources. A corporation that makes water pumps hires a lot of meat puppets to build water pumps. Likewise, a company that makes a social networking site hires a lot of meat puppets to sell you ads. Having a bunch of employees and being successful doesn’t mean that you did it right, though, or that any of those employees are the IT staff we’re talking about.

The one thing these companies do have, though, is automation and standardization, which are central tenets of cloud computing no matter what software you run. They automate and standardize to cut unneeded staff, reduce human error, and refocus the staff they have on the interesting problems. Ask a Rackspace employee sometime what their server to admin ratio is. It’s thousands to one. Then consider that the average enterprise is lucky if they get to double digits of servers per admin. Facebook, Google, Amazon, and Rackspace don’t have more people than they need, and they don’t have people doing things that are best left to machines and scripts, which is exactly what my second main point is. Better software, fewer humans, more clue.

Comments on this entry are closed.

  • Nice post. I think you’ve made your point quite effectively.
    If you’re building a 50K – 100K host cloud, chances are that IT infrastructure IS your business and competitive advantage. Of course you’re going to self-build that bad boy and hire some smart folks to customize the heck out of it.
    For the rest of the world, 100-1000 hosts is you run-of-the-mill IT shop just trying to keep the widgets flying out the door — and switching from packaged IT solutions with controlled risk – to something with unknown risks, saving a nickel to risk a dollar, doesn’t make much sense.

  • VMware won’t be much of a business if the only people deploying it are 100-1000 host companies. And that’s a price sensistive market and OpenStack could easily replace VMware – hence why VMware is moving to add value to their core via a widely diversified software toolset.

    They have good products, but Microsoft/Linux shows that you can’t beat Freedom & Openness.

    • A 1000 host company is a lot of licenses, man, if those are dual socket boxes it’s like $6m+ in up-front licensing. Or do you mean VMs? By ‘hosts’ we always mean the physical host running ESXi, VMs or ‘guests’ might be what you mean, which would make more sense.

      Up until recently it’s been the medium & large enterprises keeping VMware alive. Everybody is price-sensitive, and as a VMware user I’m glad for the competition because I’ve been able to get that better toolset at better prices.

  • I can concur with Bob on this one. Large enterprises won’t even look at open source products as a ‘production’ solution until they can buy a support contract with it. To us, a support contract through a third party to support an open source platform isn’t the same. Until we can buy a contract for OpenStack from OpenStack it won’t even be considered.

    It’s annyoing. There’s a lot of value in the open platforms and we likely have enough staff currently to run it successfully. The problem is the support. VMWare will gladly sell us a huge support contract. The business isn’t willing to plunge into something that they can’t purchase a true support contract on.

    It’s an unfortunate case of “it is what it is”. Large enterprise won’t adopy open source technologies. Places like Rackspace and AWS can just because there’s a significant price point there for them. Businesses (Enterprise companies) that aren’t purely AWS or Rackspace won’t move for it (in my opinion).

    • Do you see this changing? maybe slowly but changing none the less. How do you see companies like Redhat, Ubuntu and OpenGamma in this context of large enterprises and support contracts for “open source products”?

  • The sticking point people almost always have with VMware is the licensing cost. Seldom do I hear/read a serious argument regarding the stability, functionality, or quality of the product. Yes, it costs a pretty penny. But it works. It works right out of the box. It requires minimal knowledge to get up and running. There is a massive support organization to help out if you run into problems. There is professional training available if someone wants to learn more about it in a structured and time efficient manner and not spend the next 6 months working on a proof-of-concept. It is easy to find someone with expertise in it when your company’s person(s) who supports your virtual environment gets hit by a bus, wins the lottery, has a nervous break down, or just ups and quits. All of that adds to its value, and thus its price (People like to get paid for their work and investors like to see return on their investment).

    This isn’t to say that OSS doesn’t have its place. But, like previously mentioned, it often breaks down to what a company’s business is. Sure, I can make exactly the wheel that I need. A wheel that has no extra cruft, a wheel that does only what I want it to do (at least what I want it to do today… tomorrow is another story). I can do that without licensing costs. However, my business isn’t making wheels. And, professional wheel makers are out there that make better wheels then I can. Heck, they’ll even fix my wheels if they break. Even better, they will sell me a wheel that I can use right this second.

    When IT is the primary business, it might make economical sense to build instead of buy. That should definitely be a consideration and looked into. The Googles and Facebooks and Amazons and Rackspaces of the world certainly are big enough to build and support just about any software (and even hardware) they want to all in-house. However, most companies can’t do that. Even very large ones can’t do it. Even other tech-savy .coms can’t.

    Up until recently, I worked for a .com with approx. 5000 physical servers and roughly 1000 or so developers. Virtualization has been an issue for many years. The company had all of the same problems: out of space, out of power, hardware refreshes were painful, deployment/provisioning took too long, etc. However, the price tag for VMware was always an issue. They became paralyzed. Rather then make a decision, they waited, and waited. They said, “Google doesn’t use VMware… What’s Facebook doing?…. What about Amazon?” They are none of those companies, only a fraction of the size, and, frankly, aren’t as smart (it’s OK, very few are… I’m not that smart but I know it). Instead of acknowledging their limitations, the opensource geeks within the company, who run Xen/KVM/fill_in_the_blank_flavor_of_the_OSS_day_virtualization_product at home, were able to influence enough of the decision makers that VMware was too expensive and that what they do at home can scale to an enterprise with thousands of servers.

    This company now finds itself with 6 to 7 year old aging hardware throughout their data centers and no large scale solution for virtualization. They are now looking at various OSS products (for about the fourth time in 3 years) that can help them out of the jam they are in. They will have to build expertise in the products, customize the products (mainly because they won’t be able to help themselves), test the products, and roll them out into production. This carries a lot of risk to it. An hour of downtime means many hundreds of thousands of lost revenue and unmeasurable future business. There will be outages that come with the growing pains. One or two major outages pays for a lot of VMware licenses and support. Should this company have just paid the VMware cost 3 or 4 years ago? Who knows? But if they had, what else could they have spent their efforts on? What else would all those developers been able to accomplish?

    That’s not to say that using VMware would get this company out of a jam without any pain. Growing pains always exist. But having a large knowledge base and support can help manage that risk. For small companies and startups, it isn’t necessarily the right choice. I understand that. But when you start getting to the point that minutes and seconds count, the stakes are raised.

    The company I’m currently working for is not as large as my previous employer (maybe 1/3rd the size). An hour of down time is still ridiculously expensive, but we’re just not as big. We are now currently looking at dropping an OSS solution and moving towards VMware. Why? Well, the stuff we built in house (when we were smaller) sucks when you compare it to VMware. The guys that built it are either gone or don’t feel like supporting it any more, and fixing it doesn’t have a dollar value attached to it that makes the business side of the company take notice. So we have an inferior product that is now holding us back. Someone is stepping up and making the decision so that we can get on with our business – rather then trying to make a better wheel.

    Again, I’m not anti-opensource. Both of these companies are Linux shops and pay zero for support (CentOS and Debian). However, Linux has a large install base and while it might be a bit challenging to find competent admins and engineers, it is much easier than finding someone who knows the guts of OpenStack. Maybe in another few years, some of the open source cloud software will be ready and the industry will have the people available to be able to support that in house. I won’t be holding my breath. I’ll believe it when I see it.

    One of these companies is profitable.