The Lone Sysadmin

I’ve spent the morning looking at the new server models from Dell, based on the Intel Xeon 5500 series of CPUs (Nehalem-EP). These things look sweet, but there are some interesting caveats. A few of my observations so far:

1. Intel has killed the front-side bus and in its place implemented QuickPath Interconnect (QPI), a competitor to AMD’s HyperTransport. It’s speed is measured in GigaTransfers per second (GT/s), and is 4.86 GT/s, 5.86 GT/s, or 6.40 GT/s per direction, which according to the Wikipedia article I linked to is 12 to 16 GB/s per direction per link. Cool, but most people are going to pick a CPU based on price point rather than link speed, given that everything in the 5500 series smokes all previous Intel CPUs anyhow.

In multi-CPU configurations the CPUs themselves have a QPI link between them that is used to share L3 cache and memory information. This is a big win in the war against processor affinity, which promotes cache hits but is a management pain. For things like virtualization, where VMs are executing on different CPUs all the time, having cache data available globally will speed things up. No matter where you execute your cached data is close.

2. There are options for 1333 MHz RAM, which achieves this through interleaving across multiples of six DIMMs. Multiples of six DIMMs combined with what seems to be a lack of 8 GB 1333 MHz DIMMs means a maximum of 24 GB of RAM at that speed. Thankfully 800 MHz and 1066 MHz RAM is available, and you can cram 144 GB and 96 GB, respectively, in a server.

One drawback is that with one CPU you can only access a certain amount of memory. For Dell right now that caps a single CPU off at 12 GB of RAM. This probably isn’t the end of the world, as most people stuffing 16+ GB of RAM in a machine will likely opt for two CPUs anyhow. It does point to where things are going with virtualization, though. If you need less CPU than two quad-cores you’re also probably running a hypervisor of some sort, so it doesn’t pay for Intel to worry about the low-CPU, high-RAM users.

3. HyperThreading is back. Might as well have it — you’ve got all this extra CPU sitting there, sleeping, and it could be doing something useful instead. If they could just implement SETI@Home in the CPUs for those truly idle times…

4. Turbo Boost is a new feature, in combination with their new power technologies, where a CPU can essentially overclock itself in certain cases. These CPUs also have new sleep states, including much deeper sleep for the CPU and RAM, so when the server is idle it consumes a lot less power. It’s also interesting that the Intelligent Power Technology lets you customize the power consumption with profiles per application.

5. All that Intel Virtualization Technology that we’ve been hearing about is in these things. Extended Page Tables, VT-c (hardware-assisted I/O), VT-d (directed I/O, or dedicated virtual I/O devices), and VT-x (FlexMigrate) are all in here and conspiring together to make VMs fly.

6. They’ve implemented PCI Express 2.0, which doubles the bandwidth of the slots. Now a 2.0 x8 slot can perform as fast as an 1.0 x16 slot. Most of us will probably only take advantage of this on our workstations/desktops, with more awesome video cards, but for some people with HPC clusters this gets important.

Dell in particular has used this rollout to also launch it’s new management software, replacing the clunky IT Assistant, and also introduce new options to the server line, like solid state disk. Overall, though, I have to conclude that no matter what vendor you use, if you’re building out your own private cloud these new CPUs are pretty sweet building blocks.

Popularity: 5% [?]

• Crabs 'feel and remember pain' suggests new study – CNN.com
  Lots of stuff feels and remembers pain, including lots of stuff we eat. Doesn't mean we should stop eating it, just means we should stop killing it slowly.

Popularity: 2% [?]

• Noah Stokes – Front End Developer and Other Crap
  "I'll pull your data asynchronously from every orifice of your server." OMFG, funny. Props to Jeremy Zawodny for the link.

Popularity: 2% [?]

Okay, so I work at a University, and occasionally I use a bathroom frequented by students who still believe in writing creative stuff on the walls. Today I see a quote:

“I have yet to meet the person who can outsmart a bullet.”

and figuring that it isn’t an original work I go looking for the source. It turns out to be the Heavy Weapons guy in Team Fortress 2, during their “Meet the Team” videos. Whoever had the idea for that is a genius. You don’t need much character development in a first person shooter, but these are just the right length, and funny as hell.

If you haven’t watched them and are looking for something to kill a few minutes they’re worth it. My favorite is the Engineer (video below), and just search YouTube for the rest:

“Tearing me a structurally superfluous new behind” definitely needs to be part of my lexicon. The Sniper has a good quote, too: “Professionals have standards: be polite, be efficient, have a plan to kill every single person you meet.”

Popularity: 4% [?]

I’ve recently been working with VMware Capacity Planner project in my organization. It’s a useful tool in proving what I already know: that the physical hosts in my data center don’t do anything. :-)

The Capacity Planner Data Manager is a software component that you install at your site on a Windows host (in my case a virtual machine). It gathers data from your hosts, sanitizes it, and relays it or stores it for relaying to VMware’s data warehouse (where it’s analyzed). One of Data Manager’s features is that it’s agentless, and will just SSH into my Linux hosts and gather what it needs.

Problem with that, though, is that it wants to log in as root. All the documentation says to have it log in as root. But on my hosts nobody logs in as root, unless there’s some big crisis happening.

So I started messing around with it. As it turns out, at least under Red Hat Enterprise Linux, most everything that the Data Manager needs to do can be run as a normal user. The few commands it needs to run that require root are ethtool, mii-tool, and dmidecode. So you have a few options:

1. Grant the user sudo rights and change the scripts that run to use sudo for just those commands. On the Data Manager machine the scripts are all stored in C:\Program Files\VMware\VMware Capacity Planner\scripts by default, and you can just edit them to use sudo. Depending on how you do your user and rights management this could be either very easy or very hard.

2. Make those commands setuid for the duration of the exercise. Doing something like “chmod +s /usr/sbin/dmidecode” will make it run as its owner, which by default is root. When the data gathering is over you can just “chmod -s” those utilities again. This is more risky than allowing sudo rights, especially if you have a lot of users on your machines, because suddenly users can change network settings, etc. or even worse, trick dmidecode into doing something as root that it shouldn’t do. In my case I have very few local users on my hosts, and they’re all fellow admins with sudo rights, so the risk is a lot lower.

Another option is that you could make these setuid, do the inventory, remove the setuid bits, and then deactivate the Data Manager daily inventory job. No problem.

3. You can also copy the Capacity Planner scripts out to the hosts, and tell Data Manager where to find them via the Options. This might open up some options, as you could mess with the permissions on the scripts, either via chmod/chown/chgrp or via ACLs, to make them setuid but not readable/runnable by anybody but your Capacity Planner user.

There are some definite pros and cons to each, and a lot of it depends heavily on your environment. Personally, I was looking for an easy way to make this work and be able to revert any changes once we were done. It’s like “Leave No Trace” camping, only with servers.

Beyond figuring out how I was going to let it run dmidecode, ethtool, and mii-tool, I did the following:

1. On all my hosts I added a user with the same username and password so that I wouldn’t have to mess with entering hundreds of separate username/password pairs. This also means I can run one command via Capistrano to add the user, and in a month I’ll run one command to remove it again.

2. I use the pam_access.so module in /etc/pam.d/sshd so I added a line to my /etc/security/access.conf to restrict where the logins could come from:

-:vmcap:ALL EXCEPT 192.168.100.15 192.168.100.16 192.168.100.17

This denies the ‘vmcap’ user from logging in anywhere but those IP addresses.

3. I instructed VMware Capacity Planner Data Manager to sanitize the data it sends to the Capacity Planner dashboard, so that IP addresses and names and the like are stripped out.

As a result of all of this I am okay with, and more importantly, my security officers are okay with having Capacity Planner running in our data center for a month.

I’m pretty new to Capacity Planner overall, and a lot of my knowledge of it is derived from dissecting the inventory and performance gathering scripts, as well as trial and error. As always, if I missed something here or there’s a better way to do something let me know in the comments. Thanks!

Popularity: 7% [?]

• AT&T To Offer No-Contract iPhones to Existing Customers Starting March 26th? – Mac Rumors
  $699 for an iPhone? Ugh. I'm guessing this is AT&T and Apple realizing that they are losing a lot of money by not selling to unlockers.
• I.B.M. Said to Be in Talks to Buy Sun for $7 Billion – NYTimes.com
  I don't think this is going to make it past the antitrust guys. And if it does I'm not sure this is good for projects like MySQL. Maybe it's different now but as of just a couple years ago IBM didn't seem to understand open-source.

Popularity: 3% [?]

“A man’s troubles can be measured by the number of keys in his pocket.” – David Plankers

Popularity: 4% [?]

The 11307th photo I’ve taken with my Nikon D80. I kinda like it.

Popularity: 5% [?]