Table Stakes for Storage Arrays

What 1987 Storage Looked Like

What 1987 Storage Looked Like (Micropolis 760 MB SCSI HDD)

I was just looking at Andreas Lesslhumer’s post about blog posting volume in the virtualization community, and it’s depressing. I didn’t blog a whole lot here last year. Why was that? Because I was writing elsewhere!

Speaking of that, the first half of my “Six Features You Absolutely Need on Your Storage in 2015” list is up over at The Virtualization Practice, wherein I outline what the table stakes are for enterprise storage arrays, get only slightly snarky about why we’re still discussing, as an industry, why & how to use flash, and highlight the good work some vendors are doing (SolidFire, Dell, and Tintri in this post, more in next week’s second part). Check it out.

You Cannot Use open-vm-tools to Customize VMs

Homer Simpson: Kids: there’s three ways to do things; the right way, the wrong way and the Max Power way!
Bart: Isn’t that the wrong way?
Homer Simpson: Yeah, but faster!

My biggest pet peeve with open source is that projects don’t ever solve whole problems. They get 60% of the way to solving a whole problem and then run off to chase another squirrel.

The most recent example of this is VMware’s recommendation to use the open-vm-tools packages that ship with modern distributions of Linux. Dumbest recommendation ever. Why? Because the project got to 60% of the solution and stopped, effectively solving no problems for anybody. From what appears to be a VMware employee on the open-vm-tools mailing list archives:

> On Ubuntu it is very easy to install the open-vm-tools, so I wonder
> how these differ from the VMware provided? Do they have the same
> features?

There are a few features missing in open-vm-tools, e.g. DeployPkg (for guest customization), Unity etc. Otherwise, the two are pretty much the same.

Um, no, they’re decidedly not the same. With open-vm-tools, ongoing maintenance of the Tools is much easier, but you won’t get a working VM when you clone a template, deploy with vRA/vCAC/vCD, etc. Turns out that people like having working VMs, with the right IPs and whatnot. :)

With legacy VMware Tools, updating is mostly broken, but we’ve been working around that for a decade so it’s figured out. And you get working VMs from a deployment! Snazzy.

DeployPkg is critical in private cloud environments, and to pretty much everything VMware is promoting lately. Could someone add it and make life easier for those of us out here actually using this stuff? Please? And maybe stop recommending this unfinished software until then?

CentOS 7 Refusing VMware vSphere Guest OS Customizations

So I just spent two hours of my life trying to get my CentOS 7 VM template to deploy correctly with a vSphere customization specification. No matter what I did it would customize the VM, then uncustomize it, essentially leaving me with the template again. I finally asked our oracle and savior, Google, and two amazing things occurred.

First, I found the answer. About three weeks ago a fellow named Jeff Burns asked this same question on Server Fault, then answered his own question five minutes later (this is often what happens to me immediately upon filing a support case). He built on something I’d seen in /var/log/vmware-imc/toolsDeployPkg.log, where the VMware Tools couldn’t figure out what the OS is and then would abort. In short, you need to make /etc/redhat-release say:

Red Hat Enterprise Linux Server release 7.0 (Maipo)

Indeed, it works, though I’m guessing someone at VMware will fix this and we’ll have to decide which string we stick with. :)

Second, as part of his question Jeff linked to my post about Preparing Linux Template VMs. Thank you Jeff! That’s really cool. In fact, I’ve just updated that post with new thoughts on cleaning up those VM templates so they’re as small as possible, based on my ongoing experiences. Go check it out.

Apple Lawsuit Over iOS Advertised Capacity

In case you hadn’t seen it, Apple is being sued over the fact that a 16 GB iOS device does not have 16 GB of space usable on it. The Verge has a good story on it, link is below.

In contrast, Macworld’s Susie Ochs has published a whiny, elitist article entitled “Apple faces dumb lawsuit over the size of iOS 8.” This link is also below if you’d like to witness the cesspool that Macworld has become.

I don’t think the lawsuit is dumb, at all. On one hand computers have never included the space consumed by the OS when listing their storage capacities. Consider that an OS installed on a PC stays fairly static over the life of the PC. My mother’s computer’s copy of Windows 7 will be the OS on there until the PC is retired, and I’m comfortable generalizing that to most consumers. Given Microsoft support policies she won’t be forced to update to a new major OS release, or rather, that upgrade will come with the purchase of a new PC. There will be some space usage fluctuation as patches are installed, but since local disks on PCs are generally measured in terabytes, or at least 3 digit gigabytes, it’s not tight. Assuming 15 GB for Windows and a 1 TB drive the consumer loses 1% of the capacity to the OS.

Apple, on the other hand, doesn’t release security updates for older iOS versions. They also don’t guarantee backwards compatibility for applications. Applications always get the latest updates, which means that because there is no backwards compatibility, and no way for a consumer to downrev an app that may have been mistakenly updated, an owner of an iOS device is forced into running the latest major iOS version if they’d like their apps to work or their data to be secure.

I’ll repeat that, because it’s the crux of my whole argument. If you own an iOS device you must keep it at the latest iOS level to be secure. There are no security updates for older iOS versions.

A 16 GB iPhone 5S has 13.1 GB of usable space on it under iOS 8, representing a loss of 18% of the advertised capacity immediately out of the box.

Add to that the fact that iOS 8 required 5.7 GB of free space to install for an over-the-air update. It needs roughly half that if I connect it to iTunes. We’ll give Apple the benefit of the doubt here and say that to continue getting updates to my phone, via iTunes, I need 2.8 GB free. Now I’m down to 10.3 GB usable, or a 35.6% loss of advertised capacity.

Last, the nebulous “other” data class that seems to crop up on iOS devices:

iTunes Display of Space Usage on my iPhone

iTunes Display of Space Usage on my iPhone

What is this stuff? App data gets associated with the “Apps” class (when I copy a 1 GB movie to VLC on my iPhone the Apps class grows by 1 GB). There’s nothing in Settings->General->Usage that indicates what “Other” is, either. I can get rid of it for a while by wiping my phone, but then it grows back. In my case it’s consuming 1.5 GB of space. Now I’ve got 8.8 GB usable space, which is a 45% loss of advertised capacity. If I want to do over-the-air updates (which I do), I have 5.9 GB usable, for a 63% loss.

That’s complete bullshit, especially since I have to stay current with iOS.

Ever since I found myself clearing space on my iOS devices for iOS 8 I’ve thought that Apple really should start treating these things as devices with firmware, more like a Blu-ray player than a computer. My Blu-ray player has separate storage for its over-the-air updates, and conducts them seamlessly after I agree to proceed. Why doesn’t Apple have separate storage for their OS and for updates? Even the addition of a hidden 8 GB system storage volume, to hold iOS updates and “Other,” would do wonders in making iOS devices more supportable, more user-friendly, and more likely to actually get the updates Apple pushes. Maybe this lawsuit will force the issue.


Tech Bloggers: Punctuation Goes Inside Quotation Marks

One of the biggest differences between writing code for machines and writing English-language text for humans in the United States is the use of quotation marks. When you’re programming a computer a set of double quotation marks indicates a string, which is an atomic entity. As such, punctuation goes outside the quotes to delimit lists and whatnot.

#include <stdlib.h>
string animals[4] = {"Goat","Sheep","Cow","Platypus"};

This is not how it works when you’re writing in the English language. Periods and commas always go inside the double quotation marks in English.

Incorrectly punctuated sarcasm: We all know how that piece of software “works”.
Correctly punctuated sarcasm: We all know how that piece of software “works.”

Incorrect: Her number is “867-5309”.
Correct: Her number is “867-5309.”

Question and exclamation marks go inside the double quotation marks, too.

Incorrect: “What the hell are you doing to that hard disk”? he said.
Correct: “What the hell are you doing to that hard disk?” he said.

Incorrect: “Don’t press the EPO button in the data center”! he screamed.
Correct: “Don’t press the EPO button in the data center!” he screamed.

Instead of a period use a comma inside the quotes:

Incorrect: “I simply asked him why he was in his underwear and there was 1000 feet of unspooled fiber on the floor”, Joe said quietly to the HR representative.
Correct: “I simply asked him why he was in his underwear and there was 1000 feet of unspooled fiber on the floor,” Joe said quietly to the HR representative.

Yes, this presents problems sometimes. Sometimes you want to denote that something is a string, or spelled a particular way, and don’t want to misconstrue that the punctuation is part of it. For me this usually happens with server names.

Grammatically incorrect: We need to fix the fans on servers “esx-1-goat-4”, “esx-1-goat-10”, and “hosebeast”.

You have several ways to tackle this without running afoul of the pedantic grammar Nazis.

Capital letters: We need to fix the fans on servers ESX-1-GOAT-4, ESX-1-GOAT-10, and HOSEBEAST.
Italics: We need to fix the fans on servers esx-1-goat-4, esx-1-goat-10, and hosebeast.
Bold: We need to fix the fans on servers esx-1-goat-4, esx-1-goat-10, and hosebeast.

A list works, too.

We need to fix the fans on the following servers:

  • esx-1-goat-4
  • esx-1-goat-10
  • hosebeast

My favorite is caps. Easy to read, gets the point across, looks like a normal paragraph. Be creative, but ask yourself how you can write it so your meaning is crystal clear to the reader.

In conclusion, y’all are really freaking smart, yet I see you all making this mistake all the time. English is just another programming language, except you’re getting humans to do your bidding instead of a machine. Consider this a compiler warning. :)

P.S. I explicitly mention the English language & U.S.A. just because I don’t know the rules in other languages. I’m guessing that in many cases the rules are the same. Leave a comment if they aren’t (or if I’ve missed something here, like always).

P.P.S. I’ve been informed — several times — that “proper” English, aka that which is spoken in the United Kingdom, has this all backwards, including using single quotes where Americans use double and vice-versa, putting all manner of punctuation outside the quotes, etc. Of course, the Americans are the ones with it backwards. :) Do what you want but in the U.S.A. it’s a grammatical error.

Minimum Vacation

Sysadmin1138 has a post today on minimum vacation policies, an interesting twist on the unlimited vacation policies many startups now have:

The idea seems to be a melding of the best parts of unlimited and max. Employees are required to take a certain number of days off a year, and those days have to be full-disconnect days in which no checking in on work is done. Instead of using scarcity to urge people to take real vacations, it explicitly states you will take these days and you will not do any work on them.

Sysadmin1138 expounds on several ways this is a cool idea. I agree. There are real benefits to forcing employees to go (and stay) completely away for a moderate amount of time. In fact, financial institutions usually have a mandatory absence policy as part of their security program. The United States’ Federal Deposit Insurance Corporation (FDIC) encourages institutions to require no less than two continuous weeks of vacation for all employees:

During this time, their duties and responsibilities should be assumed by other employees. This basic control has proven to be an effective internal safeguard in preventing fraud. In addition, such a policy is viewed as a benefit to the well-being of the employees and can be a valuable aid to the institution’s overall training program.

If someone is out of the office for two weeks there is a greater likelihood that fraudulent activity will be visible . Less maliciously, when someone is gone and unreachable holes in training and documentation become pretty clear.

LumberghThe Achilles Heel of all these new-fangled vacation policies is that they need managers and company leadership to be results-oriented. That means corporate middle management actually has to manage people, might have to have some difficult conversations about performance, might have to occasionally deny vacation based on performance, and will definitely have to stop treating employees like kindergarteners. Just because Alice took 22 vacation days doesn’t mean Bob gets to take 22 days, if he’s not meeting expectations.

Of course, it’s hard to tell Bob that, though, especially if management doesn’t have a good framework for setting goals and expectations in the first place. And creating that framework sounds like work… isn’t it just easier to just go ahead and do things the way they’ve always been done, and leave these strange vacation pipe dreams to the Californians? And where is your TPS report?

A New Hope^H^H^H^HLook

Once in a while you’ve got to pull the trigger and actually ship some code.

It’s been seven years (!!!!) since I did anything serious with the way this blog looked and worked. There were plugins that weren’t supported anymore. The old theme had been so extensively customized by me that it wasn’t upgradable, and didn’t really work well with new functionality or WordPress releases. A lot of the new functionality duplicated what I’d hacked into the theme, too. Stuff like Google +1, sharing, etc. Plus I wanted SSL (and not just CloudFlare’s poser Google juice SSL crap, I wanted the security).

I started redoing the site six months ago, where “redoing” meant the cloud-esque “completely starting over.” The synchronization of content between my development site and the live site was pretty involved, and I found myself avoiding writing because I knew it’d just be more to synchronize later. Realistically a few more posts wouldn’t have mattered much, but it became a serious mental obstacle to writing, on all fronts.

Then came some asshats, hammering the old blog’s XML-RPC interface, trying to break in.
If you’ve tried to get to this site in the last couple of weeks and it’s been down it’s because those idiots went from breakin attempt to DoS, essentially causing Apache to use too much memory. The cognitively disabled Linux out-of-memory killer then kicked in, nuking the VM. Thanks, OOM Killer, you’ve been very helpful. Double thanks, jackhole at in the Netherlands. May you rot in hell.

Yes, I could have tuned Apache and Linux better, but the new site is nginx on CentOS 7 and I didn’t want to waste more time on old, tired web servers. So screw it, I pulled the trigger. This battle station is fully operational. Let’s do this.

Keeping My Blade Options Open

One of the types of advice I really appreciate is that which helps me to keep my options open.

I have a team from Dell in the office this week, configuring a giant pile of equipment we bought. The equipment includes a bunch of blade servers. We’ve relied on rack-mount equipment for decades, but with a push towards a private cloud we opted to jump into the early 21st century with blades. I’ve had relatively little experience with blades so it’s nice to have more experienced people around.

When I’m designing a system I always try to figure out what it’ll need to look like four years from now. Seeing the future is the hardest part of designing a system. Will our workloads increase? Will we need more CPU or storage? If we need more CPU or storage, how will we add it? Should we buy it all now anticipating the need or leave unused drive bays in an array to fill later? Will the drives be the same as the ones we have now? Will we have enough ports on the switches to grow? I even worry about things like whether the naming scheme will be flexible enough to handle growth. To compensate for this I try to give myself options, empty disk bays and RAM sockets, use 10 Gbps NICs instead of multiple 1 Gbps, etc. I’m pretty good at it with rackmount systems. Not so much with blades, yet.

Today I took our pile of Dell PowerEdge M620 blades and spread them between the M1000e blade chassis we have. The M620 is a half-height blade, so I stuck them all nicely & logically in the top row of slots, 1 through 8. I thought it looked great until a Dell guy, Tim, asked me if I’m ever going to put a full height or quarter height blade in the system.

My response, understanding immediately: “Oh, crap.

Dell-Blades-Good-LayoutThe truth is that I don’t know if we’ll ever need a full height blade, like the Dell PowerEdge M830. Likewise with the quarter-height PowerEdge M420 and M430s, which sit four-tall inside a full-height “sleeve.” By filling the top row of the M1000e chassis I prevented myself from easily being able to add either kind. It’d be better to fill top & bottom evenly from left to right. And with a two minute change, suggested in passing by someone who has way more experience with blades than me, I kept my options open for the future.

Here’s a question for you folks: is it cheaper to make a bad or ignorant decision now and pay for it later when it needs to be fixed, or pay for the knowledgeable consultant up front so you don’t waste time and money later?

Stick your thoughts in the comments.

The Right Way, The Wrong Way, and The Way It Is

I hate purists.

You know the type. They’re in all IT shops, in all projects. They’re the people who won’t do any work unless they know exactly how it’ll all look in the end. They research, endlessly. They’re pedantic. They sit and poke holes in your work, claiming that they’re just playing Devil’s advocate. They rarely start an answer with “it depends,” opting instead for condescending phrases like “if I were you” or “if it were up to me.” And they wouldn’t know a minimum viable product if it bit them in the duff.

Nobody knows how a project or product is going to look in the end. And even if you do have a great vision, nobody really knows the path that you’ll take to get there. Why? Because by actually taking action you learn things, and with that knowledge you’ll usually decide to change your course.

At least most people do, on most worthy projects. A purist doesn’t actually ever do much beyond being a corrosive asshole, though, so there’s really no need for them to change their mind. They’re not actually on a path. They’re the organizational equivalent of an airliner’s drag.

Action creates knowledge. Knowledge illuminates the way. Sometimes you’ll discover you’re on a suboptimal course, but getting to the right path requires more time or more money than you have. So you do the best you can with what you’ve been given. Some call it “pragmatic.”

Is it the purist’s “right” way? No.

Is it the wrong way? No.

It’s just the way it is, for now, for those of us that get things done.

Google Chrome Missing The URL in the Address Bar

Chrome Origin Chip in Address Bar Omnibox

The latest Google Chrome update changed the omnibox/address bar so you cannot see the URL by default. I, for one, hate it. I’m not being curmudgeonly[0], I copy URLs all the time and it’s just another step for me to click on the “origin chip” (as it’s being called) to see the URL.

Here’s how to change it back:

  1. Go to chrome://flags/#origin-chip-in-omnibox (you’ll likely have to cut & paste this in)
  2. Pick “Disabled”
  3. Click the “Relaunch Now” button at the bottom.

Rejoice in having Chrome the way you like it.

Chrome Origin Chip Gone in Address Bar Omnibox



[0] Yeah, I know, I am. GET OFF MY LAWN.