System Administration

Uncertainty

by

“Hey, do you have a second to look at this thing I wrote?” No. Not really. A “second” is at least 15 minutes by the time I get back to what I was doing. I do respect that people want to use me as a sounding board so I make time. Besides, I’m sitting here waiting for a data conversion between PostgreSQL and Oracle to fail, just watching it. “What is it?” “It’s a system to graph the performance of our servers.” “Um, we have one of those already.” “Yeah, this one is cool, though.” “Oh, what does it do differently that’s nicer?” “It polls every 10 seconds.” “It’s really interesting seeing graphs of performance that aren’t averaged over five …

Read More

VMware Workload, Environment, and I/O

by

If there is one thing about the intarweb that really, really sets me off it’s when people reply privately to public inquiries. I usually see this in list archives where someone posted “Hey, I’m seeing a certain problem, has anyone else? Email me off the list.” Hell no. You posted your question publicly, let the response be public, too. Now I have the problem and I have no idea what the solution was, because you kept it private. Why the person replying to you replied privately is beyond me, too. The internet is a big conversation, you idiots, and you’re whispering. If you comment on a post here there are three things that might happen: 1) Nothing. I don’t have …

Read More

VMware I/O Problems

by

I had previously mentioned I/O problems I was having with my VMware ESX Servers. I didn’t really elaborate on the problems I was having, but having been asked, I will. This was happening to me using ESX Server 2.x against EMC CLARiiON CX700s. It also appears to be a problem under ESX Server 3. The problems I was having were caused by logical block addressing (LBA), a feature of the PC BIOS that reworks the disk geometry so that a disk always appears to have 1024 tracks/cylinders and 63 sectors per track/cylinder. It’s a hack around some old limits on PC hardware. On storage attached to PCs the first cylinder is the master boot record (MBR), partition tables, etc. Because …

Read More

Four Things About Changes

by

I’ve been watching people make changes to servers, OSes, networks, storage, applications, and the like for a while now. I’ve even been one of the people making the changes. There are four properties that every successful change I’ve witnessed has had. 1. The change is atomic. If you are making changes to machines or systems each change should start and finish before you make the next. This keeps the system’s state consistent for other changes, and makes it easy to find what went wrong. If you are making changes to machines or systems each change should start and finish before system maintenance processes, like backups, occur. This way the system’s state is consistent for those processes. 2. The change is …

Read More

VMware L2 cache

by

The other day I had a del.icio.us bookmark for a paper from Dell on VMware ESX Server performance on Dell 2850 and 6850 hardware. If you read it take note of the tests with LAMP stack servers. A 6850 with four 3.33 GHz CPUs with 8 MB of cache could only support 2 more VMs in their benchmark than a server with four 3.66 GHz CPUs and 1 MB of cache. That’s fairly interesting, because CPUs with a lot of cache on them are usually quite a bit more expensive. This seems to imply that L2 cache is not a big factor in VMware ESX Server VM performance, at least for that particular LAMP workload. In this case the difference …

Read More

Landslide

by

I love building services that are fast and stable. It is a tribute to the service that lots of people start using it. Lots of people using it, all at once, sometimes overwhelm it, making it not fast and stable. You’re chugging along and wham! Suddenly you’re flying down the mountainside in a landslide. This is where the last three weeks of my life have gone. … In this particular case it’s a mailing list server. One of the places I work is a large university, which many of you probably have figured out by now (I also have my own consulting business, which is also probably pretty obvious). This university, which is probably pretty obvious but shall remain nameless, …

Read More

Hold, Fold, Walk Away, Run

by

There was a post today over at 37signals about sunk costs that I’ve been thinking about. It’s been rolling around in this skull of mine all day. If you aren’t familiar with the idea of sunk costs, Wikipedia states that they are costs that have already been incurred and which cannot be recovered to any significant degree. The Economics Web Institute has a good definition, too (search on that page). I was first introduced to this concept by an old boss who was studying for his MBA. “You have to be able to walk away from sunk costs,” he said. So many people consider walking away a waste of time, money, and effort, though. In order to not waste their …

Read More

Little Stuff

by

I’ve been pretty burnt out lately. Lots of stuff going on and very little of me to go around. And I think I’m getting sick. When I get this way I start feeling like a worthless piece of crap because I can’t get anything done. This is what Joel Spolsky was talking about in his “Fire and Motion” post. Like he says, the trick is just getting started, on something. So I started on little stuff. I upgraded our Tasks Pro installation to version 1.6.2. The latest version will check an IMAP or POP3 mailbox. Our copy of Tasks Pro has its own VMware virtual Red Hat box so it was painless to install an IMAP server, get sendmail listening, …

Read More

How to Write a Good Script

by

There is definitely an art to writing scripts that don’t suck. It isn’t enough to just get the job done. It’s everything to make your script do the right thing, with messages, with errors, and over time (like years from when it was written). 1. Always use absolute paths for everything. You cannot assume what your environment will be. You can’t. If your script executes via cron it’s likely it won’t have any of the environment variables you depend on. Including PATH. Or HOME. The working directory will probably not be what you expect, either, so don’t write to files in the directory you’re in without thinking about it. Especially if you’re running as root. You’ll actually have permission to …

Read More

Dell Firmware Updates

by

For posterity, if your Dell firmware updater keeps telling you: An Update Package is already running. Wait until it is complete before proceeding with another update. it’s /var/lock/.spsetup.

+ More Posts +