The Lone Sysadmin

Shameless Self Promotion – Active System & OpenStack Edition

Bob Plankers — Fri, 10 May 2013 07:49:25 +0000

I’m continuing to write over at The Virtualization Practice, and it’s been fun so far. Those of you following what I’ve been doing have probably seen me take a real turn towards converged infrastructures in the last six months, both for TVP and for TechTarget. Not that I don’t think the public cloud is attractive to many, but hardware vendors are doing some real interesting things that are keeping on-site IT fairly attractive. Plus the local telco lobbies and myopic/dirty legislators seem to be keeping inexpensive bandwidth, the Achilles heel of the cloud, to a minimum in most non-urban places. Anyhow, we’ve got:

A Look at the Dell Active System 800

wherein I’m trying to figure out if Dell’s converged anything worth using (I think so, especially if you need physical deployments or want respectable amounts of RAM on your physical nodes). I largely ignore their vStart lineup, which is all basically the Active System without as much standardization and integrated management, but hey, I don’t have infinite time here, and vStart basically just converges how you buy a bunch of gear. Not that vStart is bad, just not what I consider convergence.

I also wrote about:

OpenStack and the Software Defined Data Center

a few weeks ago during the OpenStack Summit as they were announcing some new, badass features in Grizzly. The rate OpenStack has been moving forward is very impressive, doing major releases every 4-5 months as they sprint to catch VMware (or pass them, if you need features like distributed tiered storage, software-defined networking, or that new-fangled IPv6 crap so you can be part of the 21st century and also talk to mobile devices, or anybody in Asia… not that I’m bitter). You still need a mad scientist or two on staff to run the thing yourself, but there’s finally enough good stuff in the base product that we’re seeing some real vendor offerings with real enterprise support at price points that are going to start making people think about using OpenStack somewhere other than their lab.

I just wonder when we’re going to start seeing the inevitable fragmentation of OpenStack, including a loss of interoperability. At a certain point all open source projects endure some infighting, or someone will have a good idea they want to keep to themselves, and the open source code means a codebase fork is the functional equivalent of taking one’s ball and going home. I hope I’m wrong, but I’d bet money it’ll happen in the next year. I’ll also bet that it won’t be VMware’s fault, though they’ll somehow end up the scapegoat. I just worry that the currently-bleak third-party tool ecosystem for OpenStack will remain that way if there’s fragmentation.

Anyhow, thanks for reading. Y’all rock.

Did you like this article? Please give me a +1 back at the source: Shameless Self Promotion – Active System & OpenStack Edition

This post was written by Bob Plankers for The Lone Sysadmin - Virtualization, System Administration, and Technology. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2013. All rights reserved.

SELinux & Return On Time Invested

Bob Plankers — Tue, 07 May 2013 06:06:03 +0000

I’m a little behind on my reading, but I wanted to address Major Hayden’s blog posts about disabling Security-Enhanced Linux, or SELinux, which brings mandatory access control to Linux. Mandatory access control is a completely different permission model for UNIX-based hosts, and Mr. Hayden feels it is underutilized:

After many discussions with fellow Linux users, I’ve come to realize that most seem to disable SELinux rather than understand why it’s denying access. In an effort to turn the tide, I’ve created a new site as a public service to SELinux cowards everywhere: stopdisablingselinux.com.

It’s pretty rare for me to argue against a security technology but in my eyes SELinux isn’t a solution to very many problems. I know how SELinux works, what it does, how to configure it and troubleshoot it, and as a result I disable it everywhere. Here’s why:

1. While SELinux ships as part of most Linux distributions, few distribution maintainers have it set up correctly for all the services that ship with their OS. Perhaps web servers work fine if you use the defaults, but deviate from those, or install another service like a mail server or tftpd and there’s a real good chance it’ll malfunction. And by “malfunction” I mean “fail silently” which just leads to a bunch of time spent troubleshooting. That’s time out of my life I’m never getting back again. On top of that, patching those distributions sometimes resets the SELinux configurations, which both annoying and a symptom of Linux vendors not caring enough to do a good job. You can deal with patching problems with more testing, but that’s also more time and effort down the drain.

2. Absolutely no third-party vendors support SELinux[0]. Which is unfortunate, as that’s one of the most compelling use cases since those vendors often have gaping, unpatched security holes in their products. As a result it’s rare that you’ll have an entire environment with SELinux turned on, which means you’ve just added yet another variable to account for in your data center (has SELinux enabled, has SELinux in permissive mode, or does not have SELinux enabled), three times as many system configurations to account for, and you should do three times more testing, too.

3. SELinux protects the interface between applications and the OS. It doesn’t inherently protect the application, beyond keeping the host itself secure. That’s important, but less so now that we have clouds and virtualization. Back when we bought one expensive server, ran one OS image on that server, and ran a ton of different apps alongside each other in that single OS image it was very important to protect one application from another. Now that we have lots of smaller OS images running in isolation inside virtual machine monitors the focus is on application security. If an intruder breaks through the application and manages to do something to the OS you redeploy the application on a fresh VM. Only this time it’s with the latest OS & application patches applied, too.

For busy IT shops and busy sysadmins return on investment is always a factor. In my eyes SELinux is a tool which will never save you enough time to justify having implemented it in the first place. Very low ROI. When it is up to me I’d rather see scarce IT time and money spent on more productive security initiatives:

Implementing regular OS patching. So few places do this well or in a timely fashion and it treats the root cause of many security problems directly. SELinux treats symptoms, not causes.
Implementing better service design, so that patching and system outages can take place without service outages, and that services might be scalable to handle additional workload from users or denial-of-service attacks.
Implementing configuration management tools, like Chef or Puppet. If time is money these systems print money. They open the door to better testing, rapid & repeatable deployments, elastic performance scaling, and good system documentation. I’ve also found they also lead to better, tighter security controls. For example, instead of adding large IP ranges to firewall rules “because it’s easier” you add just the specific IPs you need. With these tools it’s not a big deal to add another IP later if you need to. It’s also possible to do configuration audits with these tools, thereby finding discrepancies, potential backdoors, permission errors, etc. Heck, proper implementation of these tools should make enabling SELinux easier, but I still would take a critical look at the overall ROI of that decision versus other moves you can make.
Implementing better application security, whether it’s code reviews, automated code testing, IDS/IPS, patching third-party apps, etc. The application is where your data lives and is often the weakest link in the security chain. Nowadays the OS is expendable, and rather than protect the OS under the application we should focus instead on hardening the application itself. You focus on intruders coming into your house via the doors and windows, right? Not through the middle of the basement floor.

In an ideal world we’d all have time to set this sort of thing up, and it would work perfectly. This isn’t an ideal world, and as such compromises are made. Don’t let someone shame you into wasting your time by calling you a SELinux coward. You’re in good company[1].

—-

[0] I have not checked all of them. It’s possible one or two do, but I would be surprised.

[1] At the very least you have company. :)

Did you like this article? Please give me a +1 back at the source: SELinux & Return On Time Invested

Adobe Photoshop Lightroom 5 Wish List

Bob Plankers — Mon, 15 Apr 2013 20:04:08 +0000

It looks like TechCrunch & USA Today blew an embargo on the Adobe Photoshop Lightroom 5 beta due to be released, and now lots of places are getting in on the action. I thought it’d be a good time to update my Lightroom wish list from version 4. The leaked posts mention a few new features that make it easier to retouch images, some automation around leveling photos (which would be a godsend for me, I cannot seem to take a level handheld vertical shot), and some catalog improvements. By & large, though, the announcements were pretty content-free, so I’ll definitely need to spin up a virtual machine to see what else they might have added.[0]

Here’s my list of what I’d really like in a new Lightroom. The tl;dr summary is: feature parity with Apple photo management products from 3 years ago, better sharing among photographers, and more granularity for metadata when publishing online.

1. Face recognition.

Google Picasa is a horrible product, designed by blind people who hate GUIs & user interface standards, for people who don’t value productivity. However, it has kick-ass face recognition. I’m saddened when I have to export my photos from one of the premier photo management tools and use a total hack of a tool to tag people. It’s still faster than doing it manually, but what would be nicer is Lightroom finally having a feature that every other photo management tool has. Seriously.

I continue to wish for that, plus the ability to then print a contact sheet with all unknown faces, so I can send it to someone and have them tell me who these people are.

2. Online collections linked to Lightroom’s internal collections.

Right now I maintain a collection in Lightroom for an event, then take that collection and create a second one for publishing online. This is inconvenient because I have to make sure that when I edit one collection I do the same to the second. Why can’t I just mark the Lightroom collection for publishing, via one of the online plugins? “I’d like this collection to publish to Flickr, please, and this other collection to publish to Smugmug.”

3. Per-collection settings for online collections.

Some of my published photo collections need different settings, like full resolution vs. 1000px, different watermarks, no keywords, some keywords, different copyrights, etc. The all-or-nothing nature of Lightroom’s online service support makes this very difficult.

4. Granular ways of stripping metadata.

Lightroom 4 can remove certain types of metadata in the course of publishing photos online, but they’re fixed options. I’d like that expanded so that I can remove keywords but leave EXIF data, like removing all tagged names from a particular published collection but leaving geotags and EXIF.

5. Atomic catalogs.

Atomic, as in “completely self-contained,” not “fission powered.” :) There are certain things that exist outside of catalogs, like online connector settings and watermarks, that can only be synced manually. When I go on the road I have to make sure to sync my laptop’s Lightroom watermark and online publishing plugin settings. Beyond that, things I publish on the road get different URLs because the online connectors don’t mesh up between my laptop and desktops.

It also means my catalog backups are inherently incomplete because they won’t have all that data in them. All this data needs to be written into the catalog itself, and stored with the photos so I can back it all up and move it around as an atomic unit.

6. Shared catalogs.

Hey, if catalogs were atomic maybe we could share them on a local network, so a couple of photographers could work together more easily. Heck, I’d just like to have my wife share a common family catalog with me, and be able to publish with the same settings to an online service. It’d be great if two people could use a catalog simultaneously.

I’m hoping that the catalog improvements mentioned so far for Lightroom 5 will include these things, and the journalists reporting on them just didn’t know to ask or weren’t told.

7. HDR processing.

Yes, I know third-party developers have tools to do this but I don’t want to spend $500 on additional tools to make Lightroom capable of my iPhone’s basic functionality. Really.

8. Panorama processing.

Again, my iPhone can do it. It’s 2013, come on Adobe. Book publishing was an obvious attempt to catch up to Apple in a particular way, maybe you could actually catch up in a few ways that are more meaningful to your existing customer base?

9. PNG support.

The lack of support for all modern web file formats is almost criminal. Why do I need to run a PNG through a workflow in Photoshop just so I can keep some graphics together in Lightroom? If I can save it in Photoshop I should be able to natively catalog it in Lightroom.

So come on, Adobe, get it together and make a lot of us really happy.

—

[0] While I’m not subject to Adobe’s embargoes or anything I also don’t want to poke the sleepy Adobe bear with my blog stick. The intention of this post is hoping Adobe will fix some stuff, not announcing a release. If you’re looking for more details on Lightroom 5 just wait a day or two for the release and try it yourself. Journalists are rarely users of the products they report on, so the best way to see what’s new is firsthand experience. :)

Did you like this article? Please give me a +1 back at the source: Adobe Photoshop Lightroom 5 Wish List

Shameless Self Promotion – VCE, TVP, and 787 Edition

Bob Plankers — Fri, 12 Apr 2013 20:09:17 +0000

My first-ever post as a member of The Virtualization Practice is up. I’m a little slow, I know:

Digesting The Latest VCE News: Vblock 100 and Vblock 200

wherein I criticize Vblocks for not having very much RAM, and attract the attention of Kendrick Coleman in the comments (which is cool, Kenny is great). I’m very much looking forward to writing more stuff with Edward & Bernd & Steve & crew.

I’ve also been writing for TechTarget’s “Modern Infrastructure” magazine as a regular columnist, which has been pretty darn different & fun. April’s work is:

The Benefits of Insourcing Data Center Operations

wherein I wonder if moving to the cloud is like the offshoring & outsourcing manufacturing companies in the US did in the 1980s and 1990s, which is biting them in the duff in certain ways now (like a certain plane numbered 787).

Did you like this article? Please give me a +1 back at the source: Shameless Self Promotion – VCE, TVP, and 787 Edition

VMware vCenter Server Appliance & NTP

Bob Plankers — Tue, 02 Apr 2013 21:13:51 +0000

If you’re trying to configure NTP on the VMware vCenter Server Appliance (vCSA) 5.1 builds 799730, 880472, or 947940 according to the official documentation you might be seeing what I’m seeing:

vcenter:~ # yast2 ntp-client add server=0.us.pool.ntp.org
Error:
Cannot update the dynamic configuration policy.

vcenter:~ # yast2 ntp-client enable
Error:
Cannot update the dynamic configuration policy.

This appears to be a SuSE bug. Seems serious but it isn’t, the commands actually do complete correctly. If you want to check the work just use the command:

cat /etc/ntp.conf

to check for lines starting with “server” near the bottom.

/sbin/chkconfig ntp on

will enable the service at boot, and

/etc/rc.d/ntp start

will start it immediately if it isn’t started.

/usr/sbin/ntpq -p

will display the NTP daemon’s peers, which are the servers you defined. If you’re using a pool.ntp.org address expect to see the real hostname of the NTP server you’re connecting to, as well as that server’s upstream source as the refid. A prefacing “+” means ntpd is using that source in its calculations, a “-” means that it’s been rejected (often too much jitter), and a “*” means it’s the “system peer” or the best source available at the moment.

Did you like this article? Please give me a +1 back at the source: VMware vCenter Server Appliance & NTP

Why Your April Fools Post Isn’t Funny

Bob Plankers — Mon, 01 Apr 2013 15:53:32 +0000

Ah yes, April 1. The day I wipe all my news feeds and wait for April 2. “Why?” you ask. Here are the general problems with the posts I’ve seen so far this morning.

A corporate prank post announces a feature in a product that would actually solve a problem for people.

But ha ha, you’re kidding.

A corporate prank post announces a feature that wouldn’t solve a problem for anybody.

What made the early April 1 RFCs a little amusing was that they relied on deep insider knowledge of networking topics and were decent original parodies in their own right. Imitation might be the highest form of flattery, but you’re ripping them off, not imitating them. Please stop. Besides, very few joke RFCs nowadays are actually funny, mainly because there’s nothing original about them. Yet another protocol over an implausible overlay network? Side-splittingly hilarious![0][1]

A prank post announces something that is completely plausible, and really isn’t funny to most of the audience.

YouTube is shutting down? In the wake of Google killing other products we like and use that may seem plausible to many. The defensedistributed.com DOH/DHS takedown page is also plausible, if it is actually a joke. These types of “jokes” damage your reputation by getting people worked up over nothing. It’s also not funny to tell someone they’re fired, someone died, etc. That’s how you cause heart attacks.

A post shows a physical, in-real-life prank pulled on someone else which is dangerous, annoying, makes a giant mess, or is super unprofessional.

A few years ago I met a guy that was fired because someone pulled an April 1 “prank” on him. He gets a call on the way into the office from a vice president that’s out at a customer site trying to make a huge sale, they need some changes made to a demo system ASAP. No problem, he’s on his way anyhow. When he gets to the office he finds that someone had rearranged his cube, booby-trapped his PC physically and in software, disabled his desk phone, and then filled his cube with packing peanuts. He didn’t get the changes done in time to be shown to the customer. To be fair, all the rest of the team and their manager was fired shortly thereafter, and this is probably an extreme example, but still. I rely on my office to be reliable in the face of business needs, and when someone threatens that I have a problem with it because it usually isn’t their duff on the line. These sorts of things tend to escalate, too, from harmless to increasingly annoying, as people try to one-up each other. Many pranks are denial-of-service attacks, and I thought we didn’t like denial-of-service attacks. Right?

I might seem like a humorless bastard with this post, but I’ll say that I don’t actually like sterile & always-professional office environments. You spend a lot of time with the people you work with and if you can’t joke and have fun it makes for a real soul-sucking experience. It’s worth thinking about the unintended consequences of pranks, though. The best pranksters do things that are very easily recovered from and don’t endanger lives or careers. Most others would be better served by retelling a joke they heard, or passing along the occasional WTF-style URL. My coworkers and I have a non-work, invite-only email list for precisely that. After all, on the Internet the cliche holds: truth is often stranger than fiction.

In the social media world it’s worth thinking about whether you’d make it as a comedian or not before you post that “funny” stuff. Since the barrier to entry is really low social media doesn’t necessarily follow the P.T. Barnum “no such thing as bad publicity” rules, especially if you waste people’s time and energy reacting to something false. Just remember, we describe sour milk as “funny,” too.

———

[0] That’s sarcasm.

[1] It’s funny if it’s over an implausible physical medium, like Token Ring. ROFLMAU.[2]

[2] Just kidding, that’s wearing off for me, too.

Did you like this article? Please give me a +1 back at the source: Why Your April Fools Post Isn’t Funny

Preparing Linux Template VMs

Bob Plankers — Tue, 26 Mar 2013 06:18:49 +0000

Dan over at Bashing Linux has a good post on what he does to prep his template VMs for use with Puppet. He’s inspired me to share how I prepare my Linux VMs to become a template. He’s got a few steps I don’t have, mainly to prep for Puppet, and I have a few steps he doesn’t have. One big difference is that I don’t prepare my template images for a particular configuration management system, but instead bootstrap them once they’re deployed. Why? I use my templates for a variety of things, and sometimes the people who end up with the VMs don’t want my management systems on them. It also means I have to handle some of what he does in his prep script via the configuration management system, but that’s just fine. I’d actually rather do it that way because it helps me guarantee the state of the system. Not saying he’s wrong, he’s got different problems to solve than I do.

You can do this in full multiuser — runlevel 3 — or in single-user by issuing an “init 1″ and waiting for all the processes to stop. I wouldn’t do any of this in runlevel 5, with full X Windows running. In fact, I really don’t suggest installing X Windows at all on VMs unless you really, really need it for some reason… but that’s a whole different topic. I’d also suggest taking a snapshot of your template prior to trying any of this out. As Lenin said, “Trust, but verify.”

Step 1: Clean out yum.

/usr/bin/yum clean all

Yum keeps a cache in /var/cache/yum that can grow quite large, especially after applying patches to the template. For example, the host where my blog resides has 275 MB of stuff in yum’s cache right now, just from a few months of incremental patching. In the interest of keeping my template as small as possible I wipe this.

Step 2: Force the logs to rotate.

/usr/sbin/logrotate –f /etc/logrotate.conf
/bin/rm –f /var/log/*-???????? /var/log/*.gz

Starting fresh with the logs is nice. It means that you don’t have old, irrelevant log data on all your cloned VMs, and it also means that your template image is smaller. Change out the “rm” command for one that matches whatever your logrotate renames files as. Also, if you get really, really bored it’s fun to look at the old log data people leave on virtual appliances. Lots of leaked information there.

Step 3: Clear the audit log & wtmp.

/bin/cat /dev/null > /var/log/audit/audit.log
/bin/cat /dev/null > /var/log/wtmp

Again, might as well clear the audit & login logs. This whole /dev/null business is also a trick that lets you clear a file without restarting the process associated with it, useful in many more situations than just template-building.

Step 4: Remove the udev persistent device rules.

/bin/rm -f /etc/udev/rules.d/70*

I have a whole post on this, “Why Does My Linux VM’s Virtual NIC Show Up as eth1?” This is how I’ve chosen to deal with the problem.

Step 5: Remove the traces of the template MAC address and UUIDs.

/bin/sed -i ‘/^\(HWADDR\|UUID\)=/d’ /etc/sysconfig/network-scripts/ifcfg-eth0

This is a corollary to step 4, just removing unique identifiers from the template so the cloned VM gets its own. Thanks to Ed in the comments for the reminder about sed. You can also change the “-i” to “-i.bak” if you wished to keep a backup copy of the file.

Step 6: Clean /tmp out.

/bin/rm –rf /tmp/*
/bin/rm –rf /var/tmp/*

Under normal, non-template circumstances you really don’t ever want to run rm on /tmp like this. Use tmpwatch or any manner of safer ways to do this, since there are attacks people can use by leaving symlinks and whatnot in /tmp that rm might traverse (“whoops, I don’t have an /etc/passwd anymore!”). Plus, users and processes might actually be using /tmp, and it’s impolite to delete their files. However, this is your template image, and if there are people attacking your template you should reconsider how you’re doing business. Really.

Step 7: Remove the SSH host keys.

/bin/rm –f /etc/ssh/*key*

If you don’t do this all your VMs will have all the same keys, which has negative security implications. It’s also annoying to fix later when you’ve realized you’ve deployed a couple of years of VMs and forgot to do this in your prep script. Not that I would know anything about that. Nope.

Step 8: Remove the root user’s shell history

/bin/rm -f ~root/.bash_history
unset HISTFILE

This good idea is courtesy of Jonathan Barber, from the comments below. No sense in keeping this history around, it’s irrelevant to the cloned VM.

Step 9: Zero out all free space, then use storage vMotion to re-thin the VM.

#!/bin/sh

# Determine the version of RHEL
COND=`grep -i Taroon /etc/redhat-release`
if [ "$COND" = "" ]; then
        export PREFIX="/usr/sbin"
else
        export PREFIX="/sbin"
fi

FileSystem=`grep ext /etc/mtab| awk -F" " '{ print $2 }'`

for i in $FileSystem
do
        echo $i
        number=`df -B 512 $i | awk -F" " '{print $3}' | grep -v Used`
        echo $number
        percent=$(echo "scale=0; $number * 98 / 100" | bc )
        echo $percent
        dd count=`echo $percent` if=/dev/zero of=`echo $i`/zf
        /bin/sync
        sleep 15
        rm -f $i/zf
done

VolumeGroup=`$PREFIX/vgdisplay | grep Name | awk -F" " '{ print $3 }'`

for j in $VolumeGroup
do
        echo $j
        $PREFIX/lvcreate -l `$PREFIX/vgdisplay $j | grep Free | awk -F" " '{ print $5 }'` -n zero $j
        if [ -a /dev/$j/zero ]; then
                cat /dev/zero > /dev/$j/zero
                /bin/sync
                sleep 15
                $PREFIX/lvremove -f /dev/$j/zero
        fi
done

This script is partly ripped off from someone on the Internet who didn’t have a copyright note in their work (and we’ve lost track of the source – if it’s yours leave me a comment), and partly the work of my team. It basically fills each filesystem to 98% of full with the output of /dev/zero, as well as creating a logical volume to zero out the unused space in the volume groups. Why do this? Well, if you storage vMotion the template VM to another array, or to another datastore on an array without VAAI, and you specify thin provisioning, the software datamover will suck all the zeroes back out of the image, and it’ll be as small as possible. Keep in mind you can’t do this within an array using VAAI, because under VAAI the array does the copying, and the zero-sucking magic is only in the software datamover at the ESXi level. Just move it to a local disk and back to your array if that’s the case. This is also cool if you have storage that deduplicates, too, like NetApp arrays.

Why only to 98%? That way you can run it on operational VMs and it lessens the chance of causing something to crash because you filled the filesystem. :) On the templates you can probably push it to 100%, just adjust the math in bc.

Keep in mind that by writing zeroes to the free space you effectively un-thin the disks, so make sure you have enough space available in your datastore.

So that’s my prep routine. It relies heavily on keeping the rest of the VM clean, and only cleans up what we can’t avoid sullying. What else am I missing here? Leave me a comment!

Did you like this article? Please give me a +1 back at the source: Preparing Linux Template VMs

RSS & Punctuated Equilibrium

Bob Plankers — Fri, 15 Mar 2013 20:45:06 +0000

All this talk about RSS & the social media ecosystem evolving now that Google Reader is end-of-life has me thinking of Niles Eldredge & Stephen Jay Gould’s 1972 groundbreaking work, Punctuated Equilibrium.

Wikipedia explains it better than I can:

Punctuated equilibrium (also called punctuated equilibria) is a hypothesis in evolutionary biology which proposes that most species will exhibit little net evolutionary change for most of their geological history, remaining in an extended state called stasis. When significant evolutionary change occurs, the hypothesis proposes that it is generally restricted to rare and geologically rapid events of branching speciation called cladogenesis. Cladogenesis is the process by which a species splits into two distinct species, rather than one species gradually transforming into another.

Punctuated equilibrium is commonly contrasted against the theory of phyletic gradualism, which states that evolution generally occurs uniformly and by the steady and gradual transformation of whole lineages (called anagenesis). In this view, evolution is seen as generally smooth and continuous. In 1972, paleontologists Niles Eldredge and Stephen Jay Gould published a landmark paper developing this theory and called it punctuated equilibria. Their paper built upon Ernst Mayr’s theory of geographic speciation, I. Michael Lerner‘s theories of developmental and genetic homeostasis, as well as their own empirical research. Eldredge and Gould proposed that the degree of gradualism commonly attributed to Charles Darwin is virtually nonexistent in the fossil record, and that stasis dominates the history of most fossil species.

Technology seems to follow the same patterns. Some low-grade gradualism that adds relatively minor features to existing products, but it’s largely stasis until the occasional disruptive technology comes along. Even Gould’s statement about transitional forms has some parallels:

“Since we proposed punctuated equilibria to explain trends, it is infuriating to be quoted again and again…as admitting that the fossil record includes no transitional forms. Transitional forms are generally lacking at the species level, but they are abundant between larger groups.”

Hybrid drives & disk arrays, network switches that do both OpenFlow and traditional switching, the Apple Lisa, the Palm Pilot… lots of examples of transitional forms out there between product lines. With RSS it’s probably best described as a particularly large extinction event, but an event nonetheless. I do feel punctuated.

My goal now, of course, is to try avoiding the transitional forms.

Did you like this article? Please give me a +1 back at the source: RSS & Punctuated Equilibrium

Uptime Is Not Something To Be Revered

Bob Plankers — Fri, 15 Mar 2013 03:58:52 +0000

Slashdot has a link to a tribute video to a Sun that was up continuously for 3737 days. That’s 10.23 years. It’s like a sequoia tree seeing the passage of civilization around it:

My thoughts on this:

The data center and infrastructure powering this machine was built in such a way as to keep this thing powered continuously for 10 years. Whoever built and ran that infrastructure was doing a good job. It’s a generalization but I bet there are very few cloud providers that can boast anything like that.
That version of Sun Solaris is reliable enough to keep operating for years without disruption. Most OSes are, by the way, even Microsoft Windows.
That particular hardware is reliable enough to keep operating for years. Factors that influence this include enough hot-serviceable redundancy built-in, a stable environment for the server to run, clean power, etc.
Given that something like 85% of downtime is caused by human error the admins of this host were competent enough to operate the host without disrupting service, or didn’t touch it much.
The workloads for this host were sized appropriately for the host for 10 years, and any errors in this regard were resolvable without a restart.
This host probably has 10+ years of security holes on it. I’m not super familiar with patching Solaris hosts but, generally, unless you restart the software running on a host they don’t pick up library updates. So even if it is patched you’d have to restart everything running on it to guarantee library security updates take effect. Possible, just not likely. The kernel itself likely has not been patched, unless there is a mechanism to load new code in on the fly (like Ksplice on Linux). The comments on the video indicate there hasn’t been much patching.
We cannot infer anything about service availability from the little we know about this system design. There are many services that do not require high availability or continuous uptime and a vendor warranty with a certain level of response might be just fine. We can speculate that the service contract on this hardware is probably expensive, though the particular economics of replacing the system vs. maintaining a service contract are unknown to us.
The people who built this system may be gone, retired, perhaps even dead. Hopefully the builders left good documentation about it so that current admins understood its role and configuration.

To me, security is the biggest problem here, because patching is a big part of defense-in-depth. Firewalls are neat but you have to punch holes in the firewall to let people use applications, right? If an application running on a host like this gets compromised it may be very easy for the attacker to compromise the rest of the system by exploiting 10+ years of kernel vulnerabilities. Game over. In the face of threats like APT1 where attackers are coming from inside your network, or even just a firewall rule misconfiguration that isn’t caught, the kernel & system software is effectively the last good line of defense on most systems. It limits a compromise to the application and not the whole host and prevents the attackers from establishing a beachhead inside your security perimeter where they can compromise other hosts from the inside. As such, it is very important for system software to stay current. Especially in an era of virtualization, where physical hardware issues are lessened and worked around with live migration and fault tolerance features. Seeing OSes outlive their vendor support is unfortunately becoming pretty common, as hardware lifespans just don’t provide a natural OS upgrade point anymore.

These guys seem pretty aware that this wasn’t an ideal situation, and I’m not picking on them. In fact, I rather enjoyed the video, because how often do you really see something like this? This is just my occasional opportunity to reiterate that this isn’t how we, as system administrators and IT staff, should be regularly doing business. We should not be encouraging our customers and employers to do business this way, either. High-uptime systems like this become serious liabilities in so many ways, from security to lack of understanding and documentation, that when we discover them we should do what these guys did: shut it down.

Did you like this article? Please give me a +1 back at the source: Uptime Is Not Something To Be Revered

What Content Creators and Consumers Should Do Now That Google Reader Is Dead

Bob Plankers — Thu, 14 Mar 2013 06:36:32 +0000

So the tech world is freaking out about the announcement that Google Reader will go offline on July 1, 2013. There’s been talk about this for a while now, along with talk that RSS is dead. This feels like the biggest blow to 141+ character social media in history. And why did it happen? I think Dave Winer and Bruce Schneier sum it up:

Dave Winer: “Next time, please pay a fair price for the services you depend on.”
Bruce Schneier: “Don’t make the mistake of thinking you’re Facebook’s customer, you’re not – you’re the product,” Schneier said [at the RSA Conference]. “Its customers are the advertisers.”

Google Reader’s biggest problem was its API. A good API leads to a lot of extensibility, but they didn’t have a way to monetize it in the face of thousands of third-party clients. Google Reader itself became a synchronization engine with a lame web UI that few people used. Faced with Twitter, Facebook, and Google+ as ways for most non-technical people to get content I understand the decision to close it down. Google is a business, a U.S.A. corporation after all, and “Don’t Be Evil” takes a back seat to “Improve Shareholder Value.”

That doesn’t mean I like it, though. I’ve been using Google Reader in some form since its creation. I wake up and read content via Feeddler on my iPhone. I’ve got IFTTT rules that push starred articles into Buffer, tweeted throughout the day. My pre-travel routine on the iPad is to sync & cache thousands of posts to read while I’m stuck in a flying metal tube, or traveling in parts that don’t have good connectivity. All this workflow needs rethinking.

So now what?

First, if you’re worried about how you’re going to read stuff in the future don’t panic. July 1 is just around the corner, but everybody who’s ever written a feed reading service has thought about this for some time now. Take Feedly, for example. They’ve been working on cloning the Google Reader API. Theoretically that means that anybody who has written an RSS client can update it to point at them instead. There’s going to be a lot of work done over the next month or two to fill this giant vacuum, and making a hasty decision about your own reading workflow now might have to be rethought again shortly. Besides, all the services that compete with Reader are slammed now because of everybody jumping ship. Now is not a good time to evaluate a new cloud-based news reading product because they’re all performing sub-optimally.

It might be a fair time to evaluate local clients, if you read news like that. I need offline synchronization and caching but many people just read feeds via a single desktop client. I might choose to not pay much for anything right now, figuring that everything is going to change in the next month or so. Milk the free trials. It’ll also be telling who makes changes and who doesn’t. The clients that don’t change probably should be looked at as being abandoned by their developers (FeedDemon, for example).

If you have servers available to you there are interesting web-served options in Shaun Inman’s Fever and Dave Winer’s River2.

Anyhow, as Ving Rhames’ character in Pulp Fiction says, “go back in there, chill them…out, and wait for the cavalry which should be coming directly.”

Second, be prepared to pay for the services you use. Google Reader was free for the same reason that Gmail is free: they’re using your data. The next service you pick won’t be the same. This also means you should start looking at what other services you use and depend on from Google. If it isn’t something that you directly pay money for, Google Search, Google+, Gmail, or any part of their core advertising business (Adwords, Adsense, and Analytics) I’d assume that it’ll be shut down in a year or less and figure out an exit strategy. This goes for Google Calendar, Google Talk, Google Voice, and Feedburner. You don’t need to act on your strategy, just position yourself for it.

Third, get a copy of your data to use while you’re evaluating new products. Google Takeout lets you export your data as an XML & JSON archive that will likely be able to be read by replacement services. Plan to sync it again when you cut the cord in a month or two. You might use this opportunity to export all your data so you have it backed up on your own terms.

Fourth, if you’re a content creator you have some work to do. Pursuant to my second point if you use Feedburner you should start to move off of it now. I undid the redirection from my blog to Feedburner a while ago so that new subscribers get my blog feeds directly, and the Feedburner users stay alive for right now. That appears to be a prudent move, though it did mean that my readership stats need to come from somewhere else. Now is a good time to figure that out. There are also replacement services that might suit you, but they’re paid. Of course, I think that’s a good thing, even if it hurts the pocketbook initially.

It also means that if you’re using any of the Google products for your blog, like Blogger or Blogspot, you should be looking at your options and setting yourself up to move in the future. If you aren’t using your own domain that should be high on your to-do list, because it means you have more control over what happens to your readers. Use DNS to always present your readers a host name that you control, even if it means paying for a higher tier of service in a blogging platform. Setting permanent 301 redirects now might be prudent, so that when Google Reader users export their data it’ll have the updated URL already in it.

I’m going to wait until Feedburner actually dies to take the last step of moving people off of it, since I didn’t use my own domain (hey, that was 8 years ago, I was new, too). This Google Reader death might fix a lot of that for me, though.

You should probably think about the other social media options available to you, and pursue them. I’ll leave this as homework for you, though the options should be pretty obvious. At the very least you should have a Twitter account that posts new content announcements so that people can follow you.

Beyond that, expect this to be a big shakeout of readers. Your subscriber numbers will drop. For people that offer advertising opportunities this might be bad, but this also might be good. The readership numbers will turn into truly active readers, and not all the slothful folks that have you subscribed in Google Reader but never check in. Steel your ego for a correction, but don’t panic, because as the market sorts things out smaller blogs might see more revenue that was previously going to the big folks with huge subscriber counts. We just don’t know yet.

Whatever you do, remember that content is king. Write good stuff and people will come to you.

Good luck!

———

Dave Winer, “Goodbye Google Reader.”
Time Tech, “Facebook: You’re Not the Customer, You’re the Product.”
Google Official Blog, “A second spring of cleaning.”
Feedly, “Transitioning from Google Reader to feedly.”
Nick Bradbury, “The End of FeedDemon.”
Shaun Inman’s Fever.
Dave Winer’s River2.

Did you like this article? Please give me a +1 back at the source: What Content Creators and Consumers Should Do Now That Google Reader Is Dead

My Kind of March Madness

Bob Plankers — Wed, 13 Mar 2013 02:51:05 +0000

I’m not much for basketball but I laughed when my friends at Solarwinds emailed me about their version of March Madness. They’ve got a Sci-Fi Bracket going on which is totally my style.

For the people that have voted already, I do believe you’re smoking crack in at least five instances:

Wintermute is losing to HAL? Seriously? They both have taken lives, sure, and they both get involved with alien intelligence (though you don’t know that until later in each of the trilogies), but HAL is just in the right place at the right time, while Wintermute makes it happen. Plus there’s voodoo ‘n stuff, and an arm that makes Cornell box-like things.
Scotty vs. Kaylee: I’ve heard “underpromise and overdeliver” way too many times in my life, and it isn’t a good way to operate. Plus he didn’t have the stones to fix the ship in Star Trek II, Spock had to do it. Kaylee is your average small-shop sysadmin, getting it done despite a shoestring budget. Kaylee by a mile.
Q is a dick. If he hadn’t been screwing with Picard in the first place there wouldn’t have been three tachyon beams ripping space & time open. You don’t get points for fixing problems you created. In contrast, Darth Vader is the #3 greatest villain of all time. ‘Nuff said.
Adama vs. 10^th Doctor: it’s pretty obvious I could have a glass of bourbon and a cigar with Adama. And by “glass” I mean “all of it.” He also changes his mind a lot, for the better. I’m surprised the Doctor’s own companions haven’t killed his righteous, self-important ass yet. Nobody likes a know-it-all.
Hiro Protagonist has a sword, motorcycle, and millimeter wave radar. Actually, that probably makes him sort of a voyeur, now that I think about it. Kinda like the TSA. Oh well, I voted for him anyhow. I think the Snow Crash antagonist, Raven, would have been a better choice, since he’s an Aleut flintknapper and carries a nuclear warhead as personal protection.

My money is on Malcolm Reynolds to win the whole thing. And if you’re looking for something to read I definitely recommend the Neuromancer/Count Zero/Mona Lisa Overdrive trilogy, the 2001/2010/2063 trilogy, and Snow Crash.

Solarwinds’ Thwack Community: Sci-Fi Bracket Battle

Update, 3/14/2013: John Herbert called me out on some of this. As Pulp Fiction’s Jules says, “Well allow me to retort.”

HAL 9000 went operational in 1997, according to the book 2001, and though it probably wasn’t built using 1997 technology (probably 1995 or 1996, given development) we’ll ignore that. I’d never thought about the time at which Neuromancer is set, and it doesn’t look like it’s very clear, but Wintermute has limited Swiss citizenship “under their equivalent of the Act of ’53.” Presumably that’s 2053, and the book is past it. Assuming that Wintermute and its brother Neuromancer have been operating for some time at that point let’s call it 2040.

In Count Zero and Mona Lisa Overdrive they tell Angela Mitchell that Continuity is basically her sibling, a higher order AI, given the biomodification Christopher Mitchell did to his daughter. We can infer that the technology that Wintermute is constructed from isn’t yet the biochips from Maas Biolabs. Let’s say that Moore’s Law holds in some fashion, given that the epoch of biochips hadn’t occurred yet. Moore’s Law actually says that there’s roughly a doubling of transistors in a CPU every two years. It was Intel’s David House that transmuted that into a performance doubling every 18 months. Let’s say performance does double, but every three years instead.

January 1, 2040 minus January 1, 1997 is 516 months, or 14.33 performance-doubling intervals (at 36 months). That means that Wintermute’s hardware is roughly 2^14.33 more powerful, or 20642 times more powerful than HAL 9000’s could have been. If we stick with the 18 month performance doubling that’s 426,114,725 times more powerful, but then then programmers probably wrote the AI in Java or something, thereby negating most of that advantage. If I were an AI the first thing I’d do is recode myself.

Anyhow, as Kurt put it, Wintermute > HAL 9000.

As for Q vs. Darth Vader, I think you’ve sold me, though I do concede that my premise that Q is a dick could apply equally to Darth Vader. I do think that Vader shows a lot more repentance, though, as Q just proves he’s a dick all the way through to the very end of TNG.

Adama vs. 10th Doctor: eh, whatever. Adama drives what is essentially a giant gun with FTL capabilities. The 10th Doctor drives an unarmed ship that can go anywhere in time and space. On the surface of that the Doctor wins, except he’s a giant wuss and runs away all the time and sulks (Christmas 2012 is a great example of that, boo hoo, poor Doctor). You put Adama in the TARDIS and there’d be some serious fixing of stuff in the universe. And whisky consumption. But we’re not judging the characters based on their rides, we’re judging them. For me, Adama wins.

You’re dead on about Daleks vs. Borg, Spock vs. Neo. Neo based solely on Matrix 1 is different than Neo based on the whole series. WTF was up with Matrix 2 and 3, lots of unresolved questions.

Kirk and Picard… I don’t know. Shatner as an actor sucks, but the character Kirk is always willing to get into a fight. Probably his Iowan upbringing, the musical “The Music Man” gets it right with the song “Iowa Stubborn.” Yes, Stewart is English, but they say Picard is from France, so the English argument doesn’t fit. My money is on Kirk here. Not Shatner, but Kirk.

Scotty vs. Kaylee… If there were an all-star sci-fi mashup like the Avengers it’d be Geordi and Kaylee running whatever vehicles they’re flying around in, for sure.

Anyhow, the tl;dr version is: you’re wrong and you’re opinions are malformed. Ha!

Hopefully that clears some of his confusion up. Hehehe. :)

Did you like this article? Please give me a +1 back at the source: My Kind of March Madness

Last Chance: 2013 Top VMware & Virtualization Blog Voting

Bob Plankers — Fri, 01 Mar 2013 21:59:58 +0000

It’s getting to be the end of the week, and voting closes on the 2013 Top VMware & Virtualization Blogs tonight at midnight.

Why don’t you take a moment to go over there and vote?

I’d be honored if you’d vote for me, but happy if you just went and showed appreciation for all the hard work the virtualization blogger community does to provide lots of free information to the IT world. You can spend a minute saying thank you, right? :)

Did you like this article? Please give me a +1 back at the source: Last Chance: 2013 Top VMware & Virtualization Blog Voting

VMware vSphere Web Client & Internet Explorer 10

Bob Plankers — Fri, 01 Mar 2013 21:54:07 +0000

Being adventuresome and/or an idiot, I upgraded Internet Explorer to version 10. I can report two things:

1. I like it as a speedy web browser, and the vSphere Web Client performance feels vastly improved over IE 9. That’s actually been one of my complaints about the web client, that it’s pokey.

2. The remote console plugins don’t work. I have tried fidgeting with a bunch of the security controls and reinstalling the Console Helper, but it continues to report “Remote Console plugin is not properly installed.” For now, IE 10 joins the ranks of Apple Mac OS X users with no console access.

I’ll post more information as I mess with it… I’m not really a Windows guy so if you have any thoughts stick them in the comments.

Did you like this article? Please give me a +1 back at the source: VMware vSphere Web Client & Internet Explorer 10

Make Your Blog Easy To Subscribe To

Bob Plankers — Thu, 28 Feb 2013 21:53:22 +0000

A quick hint: if you want people to read your writings on a regular basis make the link to your RSS or Atom feed dead simple to find. Somewhere that doesn’t require the user to scroll to see, uses the well known, industry-standard RSS icon that is freely available, and has the word “feed” in the link.

I run into this problem often, especially with blogs hosted on Blogger or Blogspot and overly fancy WordPress themes that emphasize good looks (and mouseover events) instead of usability. For example, I just was reading a post from a blogger who has a really nice personal website, with a blog integrated into it. There’s absolutely no link to the feed anywhere. I searched the page for the word “feed” and got nothing. I searched for “RSS” and got some “Share This” widget under the posts, which isn’t what I wanted to do. Since the blog is at sitename.com/blog/ I guessed that the feed is at sitename.com/blog/feed/, popped that into a browser, but that’s some sort of comment feed.

My willingness to play cat & mouse with content is low. I gave the author a minute of my time by reading what they had to say, and I’m willing to spend 10 seconds more adding them to Google Reader. But that’s about it, because I’ve got stuff to do. In this case the blogger’s post was a link retweeted by a friend, so it’s probable that they have decent content, but they lost the one opportunity they had to convert me to a subscriber. I gave up and moved on.

It doesn’t matter how much or how little design you have if it doesn’t help people do what they need to do. If you want people to read your stuff on a regular basis make it dead simple & obvious for people to subscribe.

Did you like this article? Please give me a +1 back at the source: Make Your Blog Easy To Subscribe To

System Administration/DevOps/Cloud and Developer Positions Open @ UW – Madison

Bob Plankers — Tue, 26 Feb 2013 06:20:53 +0000

If you’ve ever thought about working with the people & organization who basically eliminated rickets by discovering how to boost and synthesize vitamin D, who took a bunch of spoiled sweet clover hay and turned it into the most popular blood thinner ever (and the most popular rat poison ever, ha!), or who isolated human embryonic stem cells so that research could happen without destroying embryos in the process, here’s your chance. The Morgridge Institute for Research on the University of Wisconsin – Madison campus is looking to hire:

as part of the Software Assurance Marketplace, or SWAMP. SWAMP is a new collaboration between the Morgridge Institute for Research, University of Wisconsin – Madison, NCSA’s Cybersecurity Directorate, and Indiana University’s Center for Pervasive Technology Institute. They’re trying to improve the security and reliability of software, particularly open-source, to improve cybersecurity. They’ve got their work cut out for them, if you ask me. :)

The development positions are looking for Python/Perl/Java/C/C++ geeks, and the sysadmin positions sound pretty hip/devops-y, too:

We are seeking candidates who have a bachelor’s degree in computer science, computer engineering, information technology or equivalent experience. At least 2 years of infrastructure/network support or related experience is desired. Candidates also must exhibit excellent customer service and communication skills as well as the ability to prioritize and meet deadlines. Ideal candidates will possess knowledge of the following:

- LAN/WAN, UNIX, Linux, Windows, Apple/Mac technology environments; Virtual server environment; storage solutions including RAID, SANs and direct attached storage
- Server room facilities like power, cooling and network distribution systems
- LAMP-based (Linux, Apache, MySQL, PHP) web server environments
- Software development environments like Git, CVS and Subversion
- Scripting languages like shell, Perl and Python
- Configuration management systems like Puppet, Chef and CFengine
- High-performance or high-throughput computing systems like Condor, PBS, SGE is a plus
- Cloud Operations Experience, especially with openstack, cloudstack, or cloupia is a big plus

Follow the links above or go to the SWAMP web site for more information or to apply. And let me know if you have any questions. While you wouldn’t be working with me (I do boring administrative computing ;) I might know a thing or two about the home of the Badgers. :)

Did you like this article? Please give me a +1 back at the source: System Administration/DevOps/Cloud and Developer Positions Open @ UW – Madison