Comments

Okay, so I discovered today that in restructuring my email I wasn’t forwarding my mail right, and therefore missing some of your comments. Ack! Sorry folks. On the bright side I did cut my spam down by about 30%. dr_justice, I will answer your question about our VMware I/O problems in a post later today. Thanks for the words of solidarity, Mark! And lastly, Jon, doing the right thing seems to fall into one of two categories: – the right thing is hard to do. – the right thing is easy to do, but is indistinguishable from the wrong thing to do, so you can’t tell if what you’re doing is right or wrong. Update: there is another category for …

Read More

Why Even Ask?

“I have some questions about the VMware environment you spec’ed out for us.” “Sure, what’s up?” “Well, you suggested a separate server for the management machine, to run VirtualCenter. Couldn’t we run that in VMware itself?” “I suppose you could, but I think you’d get some chicken-and-egg problems by having the management box inside the managed infrastructure. Personally I think it’s worth the extra money to just avoid them.” “So yes, we can.” A big difference between me a year ago and me now is that if people are going to do dumb things against my recommendations I don’t put up much of a fight. My trying to convince them to do the right thing just made them upset with …

Read More

links for 2006-05-19

Amazon.com Industrial & Scientific: screws, and nails Amazon.com: Norpro Egg Rite Egg Timer: Kitchen & Housewares popurls.com | popular urls to the latest web buzz WordPress theme park MySQL Cheat Sheet I’m totally going to build one of these for PostgreSQL PhotographyJam – the photographer’s resource: Camera basics: shutter-speed, aperture and ISO Cork’d – Welcome LeapFish.com Domain Name Resource Center

Account Reps

Why is it that when I get a good account rep they leave after 3 months, but the crappy ones stay forever? ARGGGGHGHGGGGGGHHHHHHHHHHH

Four Things About Changes

I’ve been watching people make changes to servers, OSes, networks, storage, applications, and the like for a while now. I’ve even been one of the people making the changes. There are four properties that every successful change I’ve witnessed has had. 1. The change is atomic. If you are making changes to machines or systems each change should start and finish before you make the next. This keeps the system’s state consistent for other changes, and makes it easy to find what went wrong. If you are making changes to machines or systems each change should start and finish before system maintenance processes, like backups, occur. This way the system’s state is consistent for those processes. 2. The change is …

Read More

…then don't get hacked!

“I need to add NICs to the monitoring machines,” he says. I always loathe greetings like this, because the customer is telling me how to do things and not what they want to do. “Tell me what to order. What do you want to support?” Well, that’s 50% better. Asking me what I want to support is a good start. Especially on these machines. The “monitoring” machines are fifteen machines scattered all over the region to monitor network links. Our network engineers use them to diagnose problems, and when they are idle they send traffic back and forth between each other to monitor link speeds and throughput. We don’t have terminal servers or KVM-over-IP out there, so changes to these …

Read More

Props to Red Hat for Updates 7 and 3

I have to give props to the folks at Red Hat for their latest update packs for Red Hat Enterprise Linux AS/ES/WS 3 and 4. In the past there was always at least one nasty surprise in the updates. One update pack nuked our named.conf files. One added the audit subsystem, and turned it on, so that ten days later all, and I mean all, of our machines had full /var/log filesystems (and you couldn’t log in). The last updates added a new ksh package that uninstalled the pdksh that shipped with the OS and replaced it with a really buggy ksh. The old pdksh wouldn’t reinstall unless you removed ksh and installed the pdksh RPM manually. Every time you …

Read More

VMware L2 cache

The other day I had a del.icio.us bookmark for a paper from Dell on VMware ESX Server performance on Dell 2850 and 6850 hardware. If you read it take note of the tests with LAMP stack servers. A 6850 with four 3.33 GHz CPUs with 8 MB of cache could only support 2 more VMs in their benchmark than a server with four 3.66 GHz CPUs and 1 MB of cache. That’s fairly interesting, because CPUs with a lot of cache on them are usually quite a bit more expensive. This seems to imply that L2 cache is not a big factor in VMware ESX Server VM performance, at least for that particular LAMP workload. In this case the difference …

Read More

Landslide

I love building services that are fast and stable. It is a tribute to the service that lots of people start using it. Lots of people using it, all at once, sometimes overwhelm it, making it not fast and stable. You’re chugging along and wham! Suddenly you’re flying down the mountainside in a landslide. This is where the last three weeks of my life have gone. … In this particular case it’s a mailing list server. One of the places I work is a large university, which many of you probably have figured out by now (I also have my own consulting business, which is also probably pretty obvious). This university, which is probably pretty obvious but shall remain nameless, …

Read More