The Lone Sysadmin

• Cool Tool: APC Universal Notebook Battery
  Hot damn, gotta get me one of these.
• Stripe Generator – diagonal stripes background tiles web 2.0 webdesign
• xkcd – A webcomic of romance, sarcasm, math, and language – By Randall Munroe
• WebCodr.com » Blog Archive » 15 Javascript Snippets You Can’t Live Without
• MySQL User Defined Functions for FNV (Fowler/Noll/Vo) Hash (Michael J. Radwin’s blog)

Popularity: 4% [?]

Alex over at Virtual Infrastructure 411 linked to an interesting study done by CA which indicates that 44% of server virtualization deployments were failures. This is interesting to me, and timely, too. Today I spoke to another organization with a failing virtualization project, the third so far.

The first group that I spoke with was just overwhelmed. They’d gone in head-first, buying blades, storage, everything. They had no experience with any of the gear and were drowning in it, while their managers were just seeing expenses but no savings and freaking out about it. The three IT guys were already way too busy and could not make time to figure any of the new gear out, much less become experts in it. My suggestion to them was to slow down and figure one out at a time, or divide and conquer, with each of the three becoming an expert in one of the new things. Or, hire a consultant to set it up for them, but in the long run it would be better for them to learn the technologies themselves.

Another group I spoke to related how complex virtualization is, and how their company was choosing to de-virtualize because of all the outages they were having. De-virtualize? Are you kidding me? Not knowing much about storage or virtualization, they relied heavily on vendors to suggest technologies which didn’t work well in their environment. For instance, they ended up with iSCSI storage which didn’t really mesh well with their networking group’s service levels, and the resulting finger pointing between the two groups ended up getting their project canned. “Why didn’t you just buy your own, separate network switch?” I asked…

The group today was on the right track. They had purchased nice large quad-CPU, dual core servers to run VMware Virtual Infrastructure. Lots of RAM, lots of CPU. Their mortal mistake was using a series of older NFS servers they had for the back-end storage. The performance was horrible as soon as they got more than a couple of VMs running. They had this carefully laid out budget and plan for implementation, but it hinged on the NFS servers, which weren’t cutting it. It isn’t surprising, to be honest. You need to support those beefy ESX Servers with more than a garden hose of storage capacity. Now the whole project was in danger. My main suggestion was to use the local storage until they could get some fibre channel storage hardware.

All three groups I’ve talked to should have gone more slowly. Start with two servers instead of ten. Start with dual-CPU boxes instead of quad. Start with normal servers instead of blades. Virtualize your less important machines first. Get some iSCSI or fibre channel storage and learn how to use it on a normal server first, before you attach it to your ESX Servers, before you add complexity. Use the free VMware Server first, and if you like it do a free evaluation of ESX Server.

Get it working well on a small scale, and then you’ll know what you need to be successful on a large one.

Popularity: 24% [?]

• DRBD – Distributed Replicated Block Device
• ~jk MySQL Proxy
• logwatcher: restart Apache after a segmentation fault | 2BITS.com – Drupal Development, Customization and Consulting
• Iometer project
• Coding Horror: Creating User Friendly 404 Pages
• Fontifier – Your own handwriting on your computer!
• THE CHUCK NORRIS FACTS – TOP 100 CHUCK NORRIS JOKES
• Your Identity Has Been Stolen: a 24-Point Recovery Checklist
• ScienceDaily: Bonuses Boost Performance 10 Times More Than Merit Raises
• Nick Bradbury: Holy Rush
  (tags: funny)

Popularity: 5% [?]

“It’s like leaving a book in a pail of water for five years, then trying to get it out in one piece.”

- my friend Kevin Kettner, speaking with me about how some organizations ignore their IT infrastructure so completely that when they finally need to do something with it they find they have to redo it all.

Also see “tip of the iceberg” and “yak shaving.”

Popularity: 5% [?]

• Stop web sites from resizing Firefox windows – Lifehacker
• Welcome to Desktopography | Exhibition III (2007)| Natural Desktop Aesthetics
• Wire Bin Composter

Popularity: 4% [?]

I got to see Sun’s Project Blackbox today, over at the University of Wisconsin – Milwaukee.

First impression: interesting idea, obvious that it’s the first attempt at implementation.

Now, my questions:

1. What would the money spent on a Blackbox get you in offsite hosting?

2. If you have the space to store a 20 foot shipping container and a chiller could you not just build a new data center in that space?

3. What keeps vandals and competitors from hijacking or severing the power, cooling, and external network connections? Is a whole new industry going to sprout up for data center trailer parks, with armed guards? It’s just a colocation facility writ large.

4. When Sun says that these things are stackable, does that mean system administrators need to sprout wings to get to the servers? Won’t that be a total pain in the duff? Space efficiency and ease of administration are mutually exclusive, in my experience (and marketers and engineers don’t have to deal with these products on a daily basis).

5. Why doesn’t this come in a 40 foot container?

6. Why doesn’t it have an integrated cooling option (perhaps in a 40 foot container)? You have to provide your own chiller, which makes the unit decidedly not standalone (beyond power requirements, obviously). Your chiller isn’t going to have the same level of imperviousness to the environment, so I see that as an Achilles heel.

7. Why don’t the racks have a better way to slide out, instead of that clumsy dolly thing? It was tough for the demoer to slide the rack out because the container wasn’t quite level. Will I have that same problem? The demoer mentioned the shipping product will have a dolly that moves in two dimensions. That sounds even worse, since I’ll have to manhandle a heavy rack in four directions, with running servers in it. I worry about the cable management.

8. Why does this thing need GPS? Does that mean it’s so easy to move that someone is going to steal it? Do we now need LoJack for our data centers?

9. Can I buy these things preconfigured, so my staff doesn’t have to spend time building and cabling them? A Blackbox preconfigured with and for VMware would be awesome. Ditto for Solaris zones. Plug it in and go.

10. Why are the power and cooling connections on the side, sticking out? Does this mean I can’t park two of them side by side? Ditto for the network connections, which are on the other side, so I couldn’t even park two side by side in opposite directions. To me it seems like these should be on the short ends, and all the connections in one place.

In doing some searching now I see Rackable has come out with their version of Blackbox, called Concentro. Now I have one more, big question:

11. Why didn’t Sun go all the way with the power savings thing and offer certain configurations where they do what Rackable does with DC power?

I do call shenanigans on Rackable’s 1200U of space since they do half-rack servers, but that is neat because it allows for convenient cooling up the center of the rack column with servers back to back. Those are proprietary systems, though, so they aren’t as flexible as the Blackbox for loading the container with whatever you need, but it also means they can address certain problems, like power & cooling, more easily. Concentro is a 40 foot container, so it’s comparable to Blackbox in 20 feet (300U vs. 266U). Like Sun they also don’t supply any sort of integrated cooling, though.

Overall, I think Sun could volley back at Rackable pretty easily with some custom, pre-loaded configurations based on DC power and integrated KVM & systems management. Hopefully they were planning to do that anyhow, and the information is scarce. But at the same time the demo left me with more questions than answers, which isn’t cool. To be fair I didn’t think of a lot of these until afterwards, though.

My conclusion: containerized data centers are the new blades. They’re proprietary, solve some customers’ problems, and introduce their own set of security and operational problems. It’ll be interesting to see who is buying them, what problems they are solving with them, and how these units are getting deployed.

Popularity: 7% [?]

Well, hello there y’all. I’m sorry my writing has been lame lately. It’s been pretty darn busy around me. We’re a little understaffed at work, and I’ve been spending a lot of time doing lighting design work for community theater shows. Spring is wonderful, but it also means I can see all the stuff on the ground that I didn’t get to last year, so fixing that has been in competition for time, too.

Anyhow, I just wanted to say thank you for sticking around. Thanks for all the comments, too, you guys. It’s awesome to know that people are out there, real humans reading this instead of search bots hammering this blog.

Popularity: 4% [?]

“Announcing your plans is a good way to hear God laugh.” – Al Swearengen, “Deadwood“

Popularity: 4% [?]