Uptime Is Not Something To Be Revered

Slashdot has a link to a tribute video to a Sun that was up continuously for 3737 days. That’s 10.23 years. It’s like a sequoia tree seeing the passage of civilization around it: My thoughts on this: The data center and infrastructure powering this machine was built in such a way as to keep this thing powered continuously for 10 years. Whoever built and ran that infrastructure was doing a good job. It’s a generalization but I bet there are very few cloud providers that can boast anything like that. That version of Sun Solaris is reliable enough to keep operating for years without disruption. Most OSes are, by the way, even Microsoft Windows. That particular hardware is reliable enough …

Read More

System Administration/DevOps/Cloud and Developer Positions Open @ UW – Madison

If you’ve ever thought about working with the people & organization who basically eliminated rickets by discovering how to boost and synthesize vitamin D, who took a bunch of spoiled sweet clover hay and turned it into the most popular blood thinner ever (and the most popular rat poison ever, ha!), or who isolated human embryonic stem cells so that research could happen without destroying embryos in the process, here’s your chance. The Morgridge Institute for Research on the University of Wisconsin – Madison campus is looking to hire: two build & test workflow system developers, one database developer, one software security specialist, and two system administrators, among some other positions, as part of the Software Assurance Marketplace, or SWAMP. …

Read More

Error While Loading Shared Libraries, Cannot Open Shared Object File

In the “I wish the Internet had an actual correct answer” category comes a question from a Windows colleague trying to build software on Linux. He asks “I’m trying to do some web performance testing and I compiled weighttp and the libev libraries, which worked fine, but when I try to run the program it gives me the following error.” weighttp: error while loading shared libraries: libev.so.4: cannot open shared object file: No such file or directory “I checked /usr/local/lib and the files are there. Do you have a suggestion?” Ah yes, a classic problem when building software. The problem here is that libev installed itself into /usr/local/lib: $ ls -l /usr/local/lib/libev* -rw-r–r–. 1 root root 435770 Feb 22 15:20 …

Read More

OpenStack, Lock-In, Support Costs, and Open Source Free Lunches

Since I posted my missive about OpenStack not being our savior from lock-in or support costs I’ve had a number of comments and discussions about it. The discussions generally start from the point of view that I’m wrong. Let’s take a look at a few of these. Also, it might seem like I’m picking on Randy Bias and Greg Ferro a little here but Randy seems like a good guy, and Greg is a friend, so there’s no animosity. Just point/counterpoint. TL;DR version: OpenStack is cool but isn’t some magic tech that prints money, open source doesn’t mean you don’t have to pay someone to support a service built on it, customized open source and custom solutions using open source don’t …

Read More

Should You Install Development Tools on Production Hosts?

Every once in a while I get sucked back into the age-old system administrator religious argument: should you install development tools on a production host? There are basically two schools of thought: Yes, if you need them. Despite the production nature of the host there are times where you might want to build software, perhaps something like the VMware Tools kernel modules on a Linux server, and having them helps a lot. They don’t take up much space and it’s nice to keep all your hosts identical. No, absolutely not. “Development” tools should only be on hosts that are for “development.” Build a software lifecycle management system to move everything from development to test to production. The presence of compilers …

Read More

Use 'for' Loops with the vSphere Management Assistant

The VMware vSphere Management Assistant (vMA) claim to fame is that it has a UNIX shell and the vSphere CLI installed, making it handy for a lot of things, and makes cutting & pasting comands real easy when it’s paired with a decent SSH client. One of my favorite ways to use it is with ‘for’ loops in the shell, to make the same change to all of my ESXi hosts. Let’s say you have a list of servers you want to make a change to, like using esxcli to set the HBA queue depth. My list is a text file I create in nano or vi (see my post on installing nano on the vMA), one host per line: …

Read More

Leave Some RAM For Filesystem Cache

This is post #3 in my December-long series on Linux VM performance tuning, Tuningmas. Many system administrators don’t realize it, but in most OSes RAM that’s unused by applications goes towards filesystem cache, which speeds disk operations. Some VM “right-sizing” tools don’t take this into account, and recommend pretty tight memory allocations which end up causing more disk I/O in the long term. Trading some RAM for better I/O performance is often a very good move, both for an individual VM and for the virtual environment as a whole. To understand what’s happening on a Linux VM and make a decision about how much RAM we should leave for filesystem cache we need to understand the ‘free’ command: total used …

Read More

Disable GUIs and X Windows on Linux VMs

This is post #2 in my December-long series on Linux VM performance tuning, Tuningmas. I remember an old story about people having performance issues on their Windows servers. It would happen intermittently, and never when the system administrator was around. Turns out it was the pretty OpenGL screensaver, which would kick in and slow everything down tremendously. When the admin was around he’d be using the console of the server, so no screen saver, and no problems. Graphical user interfaces make one-off administration tasks easier in some cases, but for virtual environments they come at a cost: additional RAM and CPU overhead. For a virtual environment you should shut them off, or install as little of them as you need, …

Read More

SSH Escape Sequences (aka Kill Dead SSH Sessions)

Did you know that when you’re using OpenSSH from the command line you have a variety of escape sequences available to you? SSH somewhere, then type “~” and “?” (tilde, then question mark) to see all the options. You should get something like: Supported escape sequences:   ~.  – terminate connection (and any multiplexed sessions)   ~B  – send a BREAK to the remote system   ~C  – open a command line   ~R  – Request rekey (SSH protocol 2 only)   ~^Z – suspend ssh   ~#  – list forwarded connections   ~&  – background ssh (when waiting for connections to terminate)   ~?  – this message   ~~  – send the escape character by typing it twice Most …

Read More

GNU Screen For Fun and Profit

My friends and fellow virtualization professionals Tim Oudin & Joep Piscaer & Paul Gear and some commenters reminded me that the perfect complement to SSH session instability is the GNU utility ‘screen’. Indeed, I’ve been using it for years, and I recommend it to everybody doing any significant work in an SSH session. As the FSF puts it, “screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.” In short, it virtualizes your interactive shell session so you can disconnect from it and reconnect to it as needed. On Linux hosts it usually comes as a package, else you can download it from gnu.org and compile it. I’ve built it on DEC UNIX and AIX in …

Read More