The Lone Sysadmin

Sorry folks for being out of the game for a bit. Last week got busy with flash floods in our data centers, projects going sideways, and the Wisconsin VMware User’s Group meeting on Friday. Some of you have commented on some of my posts and I want to get back to them with a response. Like a flash flood, though, I find myself being swept away in time by dozens of small fires that need extinguishing.

The VMUG was at The American Club in Kohler, WI. It’s the only five-star hotel in the Midwest, and it was really awesome that Fujitsu sponsored it. Even though it was 90° Fahrenheit outside the beer was cold. There was lots of news about the VI3 infrastructure upgrade and a lot of hate for the new licensing scheme. Plus I’m in love with the 2.2 pound Fujitsu Q2010. Hell yeah. The tablet notebooks look pretty cool, too, but I don’t have a good use for them (and they’re way more clunky than a pad of paper). Maybe if there was a theatre lighting design program for a tablet PC…

After the event my friend Derick and I decided to investigate the alcohol options at a five-star hotel. If you want to avoid some snobbiness from the 60+ year old wealthy dinner crowd[0] head for the basement, to The Winery Bar, and sit at the bar itself. The bartenders are great when you treat them like human beings, and the bar stocks just about any [good] alcohol you’d want to drink. If your bartender happens to be a guy named Brandon he knows a lot about whiskey, tequila, and mezcal. Oh, mezcal, how I adore you. I should have had more water. OMFG, my head.

Anyhow, I’m going to go play Civilization 4: Warlords for a while and try to purge my mind of work. :-)

[0] Our major offenses were that I was wearing a ball cap (I always wear a ball cap), and Derick had shorts on. We were not the most underdressed in the room, but Derick did hear a “that man isn’t wearing long pants” remark whispered behind him. I hope when I’m that old I have so little to worry about.

• Effective Style with em | Monday By Noon
• SMOKE CRACK NOT SALMON [10221] – $14.29 : DURKL t shirts: get with us. ill, suave, cool 80s t-shirts. condoms not included.
• I [heart] WordPress — yukei.net
• FictionPress.Com Column : Tools of the Trade
• Writing words vs. writing software – Signal vs. Noise (by 37signals)
• 38one Denis Radenkovic’s Portfolio: WordPress logo

“Hey Bob, how do I get PHP to connect to Oracle?”

Dude, it’s called Google. I’m a sysadmin, not development support. Oh, wait, yeah, I am development support. But not like that. Geez, get a damn book or something.

“I have a chunk of code I can give you as an example if you want.” Trying. hard. to. be. nice. because. you. are. a. knob. and. I. am. busy.

“Oh, I have code that has worked in the past. It just doesn’t work now. I think you’re missing the Oracle part on those new virtual machines.”

Why didn’t you just say that to begin with, dick? Cut to the chase.

“You’re right. Those don’t have the Oracle stuff built in to PHP. Did you say you needed it?”

“I don’t know. But I do need it.”

I know. He didn’t ask for it. He needed development, test, and production environments with a web server. That’s all he said.

“Okay, I’ll put it on my to-do list.”

“Cool, whatever. Thanks.” I am way too cool to converse with sysadmins.

At this point I chucked it into my to-do list for later. Current backlog: ten business hours. Two hours later, though, this guy shows up in my cube. I’m on speakerphone working with another team on a bunch of problems that are preceding a big server upgrade we’re doing at 5 PM.

“Hey, can you put them on hold or something?”

Excuse me? Maybe you could fuck off or something. Now is NOT the time.

“Um, we’re in the middle of a bunch of stuff. Send me an email.” Trying. hard. to. be. nice.

“How’s the Oracle thing coming?”

“I haven’t looked at it yet.”

“What do you mean by that?!?”

“I have become entangled in an upgrade that has decided to go sideways because of high-order ASCII characters that are sent in spam. I’ll fix your Oracle connectivity tomorrow and shoot you an email.”

“I’m leaving on vacation tomorrow. I told the customer it would be ready this afternoon.”

Thanks for including me in your plans, dickweed. It’s 3 PM, I have two hours to get this upgrade fixed up and go get my car from the shop. You could have also been a little less terse earlier and warned me that you were going to shit all over my afternoon. Alternately you could have asked for what you needed originally. It’s only by luck that you have PHP installed, but no surprise that it isn’t what you wanted.

“When are you leaving today?”

“I’m leaving now.”

“Okay, so I have a week to get this done.”

“No, I am going home and I’ll do it from there. It’s that important.”

“Neat.” I want to go tell this guy to cram it up his snooty ass. “Why don’t you go home and IM me and I’ll see where I’m at with these other problems.”

He starts to say something. I raise my hand to indicate that I’m done. Talk to the hand, bitch.

“Go home, IM me from there,” I say.

Then I remember that I didn’t mute the speakerphone. Oops.

“Geez, is that how developer X treats you?” asks my colleague and friend on the other end.

“Yeah,” I reply. “Ain’t it cool? He asked for that stuff, no rush, two hours ago.”

“So what’s the difference between our problems and his? These are a rush, too.”

“The difference is that you’re not going to flip out when I say that I have to go get my Jeep from the shop in a couple minutes.”

• Airfare Predictions, Find Cheap Airline Tickets – Farecast

It used to be troublesome when someone needed 200 GB of disk space. It was this big negotiation between a system administrator or storage administrator and the DBA, or the user, or the application admin about why and how and how long and space is expensive and etc. etc. I believe the right term for it is “goat rodeo.”

With the advent of 300 GB fibre channel drives and 750 GB SATA drives storage administrators don’t need to worry about any of that crap anymore. They don’t even bat an eye at a 500 GB space request because it isn’t a problem anymore. Some of you will say I am spoiled in the environment I’m in, but it’s a fact for me. You want 500 GB? Sure thing, it’ll be ready in a minute.

The problem now is I/O.

A single drive has a single set of read/write heads mounted on a single armature, which means that it can only do one thing at a time. The advent of caches and command queues means the drive can make better decisions about what order it does things, but when it comes down to it each drive has a finite amount of I/O it can do in a second. The number of IOPS a drive can sustain depends on a number of factors. It’s less with SATA, more with fibre channel. Rotational speed (RPMs) has a lot to do with it, too. A 15K drive has 5000 more opportunities a minute to do stuff than a 10K drive does.

Most users of applications want their application to complete its I/O as fast as possible, because they’re waiting for the results. Since a single drive usually cannot keep up with your enterprise demands you gang a bunch of drives together and spread the I/O across them. And what happens is you end up with a ton of space but not enough I/O capacity.

So what do we do?

You could keep adding more drives, but drives cost money. They use power. Their capacity causes licensing issues on storage arrays, as most vendors charge per terabyte for their software licenses. As a mechanical part drives fail, too, which means you have to replace them. You can take the MTBF of a drive and calculate how many hours a week you will spend dealing with dead disks. For large implementations that calculation is a very real number. Some of the large storage implementations, like those attached to the TeraGrid, have designed their systems to tolerate failure because of the staff time required to change disks. There are certain limits to adding drives to arrays, too. You have to account for rebuild times when a drive fails. The more drives you have in an array the longer the rebuild time, which puts you at risk for a second disk failure. Two drive failures in RAID5 is fatal.

You could get drives with a faster rotational speed. However, faster drives cost more money. They generate more heat, which means they consume more data center resources in power and cooling. In my experience they also fail sooner because they are less tolerant of environmental fluctuations.

You could get smaller capacity drives to add spindles but not incur licensing costs. However, as you grow you’ll end up with a lot more drives. You could also switch from RAID3-5 to RAID 10, though. As disks come down in price it is now feasible to use RAID 10 which has advantages in both I/O capacity and fault-tolerance for certain applications like databases. Storage admins just shied away from it mainly because it has been costly to waste 50% of the capacity of an array. Nowadays, though, it isn’t about capacity, it’s about I/O.

2.5″ disks show some promise in these areas. You can now get more spindles in 1U, they run cooler and consume less power. Of course, running cooler and using less power is offset by the additional drives you’ll add, so effectively you get more IOPS for the same power bill.

Another option is to manage hotspots on your disks. Inevitably when you add drives you don’t get the optimal host-I/O to I/O capacity mix. You can tweak that, though, and spend years sitting and watching performance graphs of your arrays. Most storage vendors have some capability to move a LUN around within an array. They just have no way for the array to automatically do it. Sure, some of them claim that capability but their software has so many limitations it isn’t practical. Right now “hotspot management” still means “large staff time expenditure.”

Solid state disks (SSDs) are an option. They are really freaking fast, but they scare the crap out of most people (probably unnecessarily) because they’re storing data in volatile RAM. SSDs also tend to be small in comparison to typical drives (100-200 GB), and because of that you need to design your systems to use them very specifically. One of my rules of system administration is that the more custom a system design is the more time it takes to administer it. That ultimately means more admin time, and on the whole it might be cheaper to just buy more conventional drives and manage it as you do everything else.

As you add I/O capacity to your arrays you also have to think about your SAN. Are you going to be moving the bottleneck to the fibre channel switches and HBAs?

So what do we do? It would be really nice if storage vendors designed a no-bullshit optimizer for their arrays. Something us admin types can just turn on and forget, kind of like the Dynamic Resource Scheduling in VMware’s Virtual Infrastructure 3. VMware hit the nail on the head — the systems know what they’re doing so let them make the call.

A lot of this functionality could also be moved to a storage virtualization device. I wrote about this a couple of days ago in a post about how virtualization engines suck.

In the meantime, however, storage admins just need to realize that it isn’t about dollars per GB anymore, it’s about dollars per I/O per second. Plan accordingly. And if you have a choice get as many 15K spindles as you can. :-)

• Marketallica » Blog Archive » Full Collection of Crazy Mini Ads
  (tags: design photo)
• Marketallica
  (tags: blogs business technology)
• Pandora Internet Radio – Find New Music, Listen to Free Web Radio
  (tags: music)
• WordPress Widgets
  (tags: blogs)
• JAVA Spine » Blog Archive » EJB basics
• Ten Steps to Developing Your Site’s Traffic at Today I . . . . . .
• webcast.berkeley | Courses | Schedule
• Default Passwords
  (tags: Security passwords)

Remember how there was this big thing about how Tivos were learning the viewing habits of their users, and the users were sure that their Tivos thought they were gay. Last.fm sure does have a strange idea of what I want. Listening to their “recommendation radio” is a great way to find new music, and I love the control where you can move the slider between “mainstream” and “obscure.” But dammit anyhow, I’d like a non-emo, non-Backstreet Boys recommendation from time to time. Geez.

So I’ve just loaded a playlist in iTunes with a lot of my old favorites, hit play, and turned my speakers off.

My plan will probably backfire, making the computers think I’ll like Enya and Yanni now. Sail away, mofos.

Brian over at stereoroid.com commented on my last post about what I want from a storage virtualization engine. Brian, I hope you don’t mind but I’d like to answer outside of the comments section. Your comments were not counterpoint, they were more in the realm of adding clarity, something which my other post may have lacked. I hope this doesn’t scare people away from making comments. :-) I really appreciate them.

I’m most familiar with EMC and IBM high-end and midrange offerings, and a smattering of whatever LSI Logic is called now (StorageTek/FastT/Engenio/etc.). I don’t know very much about HP EVAs. From what you’ve said it sounds like they are very much like EMC and IBM’s high-end offerings where there are no SPOFs.

The problem is that high end arrays are expensive and stodgy. Midrange arrays have cool new features on them. High end arrays are stable. Midrange arrays aren’t so stable. From what I’ve seen I’ve concluded that what makes midrange arrays cool and modern is also why they are not stable (maybe this is just a property of the arrays I have been subjected to). They try to do too much to please people, and end up pleasing nobody because the faults are so devastating. To fix problems we always need the next version of the code. To get there we need to take an outage. From my experience with EMC CX700s you will get an outage even if they say you don’t need one, so it’s best to just take one and not have a mess on your hands. My experience with StorageTek D-series arrays, aka IBM FastT, is similar in that it’s just easiest to come down. Ditto for Apple Xserve RAIDs on the low end where there is no redundancy at the controller level. Since my DR copy is within 2 kilometers I could just use that, and take the outage on the array and not my servers.

What if storage vendors split their software development from their hardware development? We know software has bugs, and that these defects are inevitable. We know that sometimes software bugs in one part of a system creep over and mess up other working parts of a system. I’ve seen this in my storage arrays, where MirrorView dumps out and kills a storage processor. What if we separated the two so that it was more clear what was working where? What if one team within a storage company set out to make an array that was ultra stable, but had only the most basic of features? What if another team set out to implement all the cool advanced features somewhere else, like a virtualization controller, at a level above the individual array? Since that’s the level we expect the software to operate at doesn’t it make sense to position it there physically as well as logically?

I’m not saying my wishlist is universal but I do think it has merit for a lot of companies and institutions. Classically, IT resources that have been considered “infrastructure” end up with very little flexibility. With that comes a general stodginess and slowness to change. These infrastructure resources are the ones with the most potential to advance, though, and while we’re marching forward I want to see us add as much flexibility as we can to them before we get entrenched again. That way we might be able to change a little faster in the future.

Interestingly enough, and as a topic change, when I talk about site failover I really am not interested in disaster recovery. If you lose a whole site you’re in trouble, and it’s a whole different mess. Not to say that I’m not concerned with it, because I am, but it’s a different problem. Some of what I propose might help with DR but I’m more interested in mitigating the ongoing week to week trouble that crops up as new applications and systems are added to a data center. I want to see administrators given more tools to fight back, and I want to see vendors thinking intelligently about how these tools are built. Cluster Extensions from HP, HACMP from IBM, all of those are really cool if you have a system that can incorporate them. I haven’t had the opportunity to work with them extensively but from the little I’ve seen they are expensive, require lots of planning, and necessarily have many rules and constraints. From a business perspective they seem to make sense but from a flexibility standpoint they leave much to be desired. I also think they are a symptom of a larger reliability problem. What if we could make monolithic machines more reliable?

Yeah, right. :-)