sysadmin

7 Ways IT Staff Can Prepare for the Holidays

by

For us IT types it is important to maintain a good balance between work and our lives. Just as they say that good fences make good neighbors, I’ve found that a good delineation between work and home improves both. The holiday season is taxing, though. People rush around trying to wrap up loose ends, they’re using vacation they’re going to lose, and they’re generally scattered and distracted, which isn’t a good thing. If you’re lucky enough to work somewhere with a true 24×7 operations center then coverage over the holidays is already thought out. However, most IT staff in the world aren’t in places like that. Here are some thoughts I have about how to defend your time off over the …

Read More

Use Microsoft Excel For Your Text Manipulation Needs

by

I’m just going to lay it out there: sysadmins should use Microsoft Excel more. I probably will be labeled a traitor and a heathen for this post. It’s okay, I have years of practice having blasphemous opinions on various IT religious beliefs. Do I know how to use the UNIX text tools like sed, awk, xargs, find, cut, and so on? Yes. Do I know how to use regular expressions? Yes. Do I know how to use Perl and Python to manipulate text, and do poor-man’s extract-transform-load sorts of things? Absolutely. It’s just that I rarely need such complicated tools in my daily work. I often just have a short list of something that I need to turn into a bunch of …

Read More

Big Trouble in Little Changes

by

I was making a few changes today when I ran across this snippet of code. It bothers me. /bin/mkdir /var/lib/docker /bin/mount /dev/Volume00/docker_lv /var/lib/docker echo “/dev/Volume00/docker_lv /var/lib/docker ext4 defaults 1 2″ >> /etc/fstab “Why does it bother you, Bob?” you might ask. “They’re just mounting a filesystem.” My problem is that any change that affects booting is high risk, because fixing startup problems is a real pain. And until the system reboots the person who executes this won’t know that it works. If it doesn’t work it’ll stop during the boot, sitting there waiting for someone with a root password to come fix it. So you’ll have to get a console on the machine and dig up the root password. Then …

Read More

System Administration/DevOps/Cloud and Developer Positions Open @ UW – Madison

by

If you’ve ever thought about working with the people & organization who basically eliminated rickets by discovering how to boost and synthesize vitamin D, who took a bunch of spoiled sweet clover hay and turned it into the most popular blood thinner ever (and the most popular rat poison ever, ha!), or who isolated human embryonic stem cells so that research could happen without destroying embryos in the process, here’s your chance. The Morgridge Institute for Research on the University of Wisconsin – Madison campus is looking to hire: two build & test workflow system developers, one database developer, one software security specialist, and two system administrators, among some other positions, as part of the Software Assurance Marketplace, or SWAMP. …

Read More

How Large Your Linux Swap Partition Should Be

by

This is post #4 in my December-long series on Linux VM performance tuning, Tuningmas. One of those timeless questions in system administration has always been “how much swap space do I configure on my server?” The old rule used to be twice the amount of memory, but does a server with 256 GB of RAM really need a half terabyte of swap? And what about VMs? Swapping on VMs is a serious performance drag. Would it be a good idea to just disable swap completely? One thing to consider is that there’s a tunable kernel parameter, /proc/sys/vm/swappiness, that controls the tendency of Linux to scavenge inactive memory pages and swap them out. It is a number from 0 to 100, …

Read More

A Little Transparency Helps

by

In my past I’ve been a theatrical lighting designer. I’m not professionally trained, I just sort of picked it up by working with community theater groups. A friend who is a professionally-trained lighting designer once gave me a tip about darkness, in one of those situations where I was trying to illuminate only a small part of the stage for a particular scene: “Always leave some light on the rest of the stage. Humans get curious about what they can’t see, and they won’t realize it but they’ll spend a bunch of time watching the absolute darkness instead of what you want them to watch.” When I light a show I usually have some general “down” light available, so the …

Read More

SSH Escape Sequences (aka Kill Dead SSH Sessions)

by

Did you know that when you’re using OpenSSH from the command line you have a variety of escape sequences available to you? SSH somewhere, then type “~” and “?” (tilde, then question mark) to see all the options. You should get something like: Supported escape sequences:   ~.  – terminate connection (and any multiplexed sessions)   ~B  – send a BREAK to the remote system   ~C  – open a command line   ~R  – Request rekey (SSH protocol 2 only)   ~^Z – suspend ssh   ~#  – list forwarded connections   ~&  – background ssh (when waiting for connections to terminate)   ~?  – this message   ~~  – send the escape character by typing it twice Most …

Read More

GNU Screen For Fun and Profit

by

My friends and fellow virtualization professionals Tim Oudin & Joep Piscaer & Paul Gear and some commenters reminded me that the perfect complement to SSH session instability is the GNU utility ‘screen’. Indeed, I’ve been using it for years, and I recommend it to everybody doing any significant work in an SSH session. As the FSF puts it, “screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.” In short, it virtualizes your interactive shell session so you can disconnect from it and reconnect to it as needed. On Linux hosts it usually comes as a package, else you can download it from gnu.org and compile it. I’ve built it on DEC UNIX and AIX in …

Read More

Keep SSH Sessions From Disconnecting

by

With the installation of new firewalls at work I’ve been getting disconnected a lot from hosts I SSH to, due to changes in the inactivity timeouts. It’s particularly inconvenient when I’m tailing a log that hasn’t changed in a while… and then the connection dies, and I don’t notice. Oops. It also happens in various other situations, like NAT through a home router, too. Let the connection sit for a while and you’ll have to recreate it. I could ask our network guys to change the timeouts, but it turns out there’s a better fix from the client side. SSH protocol version 2 supports server keepalive, essentially NOOPs sent to the server to keep the connection up. You can do …

Read More

Cargo Cult System Administration

by

The term “cargo cult” comes from World War II, where the USA and Japan created small airstrips and bases on islands in the Pacific Ocean, and the locals mistook the cause & effect in relation to why goods and cargo appeared on the island. It’s absolutely fascinating, and the Wikipedia entry for cargo cults indicates some of these cults are still operating. The excellent book Code Complete, by Steve McConnell, introduced me to the idea of cargo cult software engineering. Turns out, a guy by the name of Eric Lippert coined the phrase “cargo cult programming,” and explains the whole thing well: During the Second World War, the Americans set up airstrips on various tiny islands in the Pacific.  After …

Read More

+ More Posts +