System Administration

The Mechanic's Car

by

It’s been almost a month since I’ve been able to post here. I’m hoping it’s like riding a bicycle, and I’ll get the hang of it again. I knew I’d be insanely busy throughout December, so I scheduled a bunch of posts (one a day) to auto-post. I figured I’d have time to tend comments when I had connectivity, but not any time to do any real writing. I’ve been very unhappy with WordPress’ scheduling mechanisms. Perhaps it’s just user error (PEBKAC, even — Problem Exists Between Keyboard And Chair). As a result I decided to delve into the XML-RPC stuff and write my own autoposter script. Something I could schedule in cron, and have it tweet some promotional stuff, …

Read More

How Large Your Linux Swap Partition Should Be

by

This is post #4 in my December-long series on Linux VM performance tuning, Tuningmas. One of those timeless questions in system administration has always been “how much swap space do I configure on my server?” The old rule used to be twice the amount of memory, but does a server with 256 GB of RAM really need a half terabyte of swap? And what about VMs? Swapping on VMs is a serious performance drag. Would it be a good idea to just disable swap completely? One thing to consider is that there’s a tunable kernel parameter, /proc/sys/vm/swappiness, that controls the tendency of Linux to scavenge inactive memory pages and swap them out. It is a number from 0 to 100, …

Read More

Leave Some RAM For Filesystem Cache

by

This is post #3 in my December-long series on Linux VM performance tuning, Tuningmas. Many system administrators don’t realize it, but in most OSes RAM that’s unused by applications goes towards filesystem cache, which speeds disk operations. Some VM “right-sizing” tools don’t take this into account, and recommend pretty tight memory allocations which end up causing more disk I/O in the long term. Trading some RAM for better I/O performance is often a very good move, both for an individual VM and for the virtual environment as a whole. To understand what’s happening on a Linux VM and make a decision about how much RAM we should leave for filesystem cache we need to understand the ‘free’ command: total used …

Read More

Disable GUIs and X Windows on Linux VMs

by

This is post #2 in my December-long series on Linux VM performance tuning, Tuningmas. I remember an old story about people having performance issues on their Windows servers. It would happen intermittently, and never when the system administrator was around. Turns out it was the pretty OpenGL screensaver, which would kick in and slow everything down tremendously. When the admin was around he’d be using the console of the server, so no screen saver, and no problems. Graphical user interfaces make one-off administration tasks easier in some cases, but for virtual environments they come at a cost: additional RAM and CPU overhead. For a virtual environment you should shut them off, or install as little of them as you need, …

Read More

Align Your Partitions

by

This is post #1 in my December-long series on Linux VM performance tuning, Tuningmas. I wrote about it back in 2006, and lots of others have written about it since (Duncan Epping has a nice vendor-agnostic post with diagrams): misaligned storage trashing your I/O performance. What’s the big deal? In short, it is killing your I/O performance. Logical Block Addressing on your disk drive makes the Master Boot Record 63 bytes long. This means it occupies sectors 0-62 on disk, and the first partition will start at sector 63. The number 63 is a persona non grata in the computer world. It isn’t a power of 2, and it certainly doesn’t line up with your storage’s idea of the world …

Read More

A Little Transparency Helps

by

In my past I’ve been a theatrical lighting designer. I’m not professionally trained, I just sort of picked it up by working with community theater groups. A friend who is a professionally-trained lighting designer once gave me a tip about darkness, in one of those situations where I was trying to illuminate only a small part of the stage for a particular scene: “Always leave some light on the rest of the stage. Humans get curious about what they can’t see, and they won’t realize it but they’ll spend a bunch of time watching the absolute darkness instead of what you want them to watch.” When I light a show I usually have some general “down” light available, so the …

Read More

SSH Escape Sequences (aka Kill Dead SSH Sessions)

by

Did you know that when you’re using OpenSSH from the command line you have a variety of escape sequences available to you? SSH somewhere, then type “~” and “?” (tilde, then question mark) to see all the options. You should get something like: Supported escape sequences:   ~.  – terminate connection (and any multiplexed sessions)   ~B  – send a BREAK to the remote system   ~C  – open a command line   ~R  – Request rekey (SSH protocol 2 only)   ~^Z – suspend ssh   ~#  – list forwarded connections   ~&  – background ssh (when waiting for connections to terminate)   ~?  – this message   ~~  – send the escape character by typing it twice Most …

Read More

GNU Screen For Fun and Profit

by

My friends and fellow virtualization professionals Tim Oudin & Joep Piscaer & Paul Gear and some commenters reminded me that the perfect complement to SSH session instability is the GNU utility ‘screen’. Indeed, I’ve been using it for years, and I recommend it to everybody doing any significant work in an SSH session. As the FSF puts it, “screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.” In short, it virtualizes your interactive shell session so you can disconnect from it and reconnect to it as needed. On Linux hosts it usually comes as a package, else you can download it from gnu.org and compile it. I’ve built it on DEC UNIX and AIX in …

Read More

Keep SSH Sessions From Disconnecting

by

With the installation of new firewalls at work I’ve been getting disconnected a lot from hosts I SSH to, due to changes in the inactivity timeouts. It’s particularly inconvenient when I’m tailing a log that hasn’t changed in a while… and then the connection dies, and I don’t notice. Oops. It also happens in various other situations, like NAT through a home router, too. Let the connection sit for a while and you’ll have to recreate it. I could ask our network guys to change the timeouts, but it turns out there’s a better fix from the client side. SSH protocol version 2 supports server keepalive, essentially NOOPs sent to the server to keep the connection up. You can do …

Read More

Why Does My Linux VM's Virtual NIC Show Up As eth1?

by

Got a Red Hat Enterprise Linux (or CentOS, or Scientific Linux, or Oracle Enterprise Linux, etc.) 6 VM that won’t bring up it’s single network interface after you clone it? Get the following error when you try using /sbin/ifup to enable it? “Device eth0 does not seem to be present, delaying initialization.” When you use “/sbin/ifconfig -a” you see eth1 where eth0 should be? eth1      Link encap:Ethernet  HWaddr 00:50:56:9B:00:85             BROADCAST MULTICAST  MTU:1500  Metric:1           RX packets:812 errors:0 dropped:0 overruns:0 frame:0           TX packets:214 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:72509 (70.8 KiB)  TX bytes:28324 (27.6 KiB) lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0           inet6 addr: ::1/128 Scope:Host           UP LOOPBACK RUNNING  MTU:16436  Metric:1           …

Read More

+ More Posts +