The Lone Sysadmin

• Inside the Attack That Crippled Revision3
  In short, a company working for the RIAA/MPAA (MediaDefender) decided to SYN flood Revision3 because they had an open BitTorrent tracker. Nevermind it was theirs, used to disseminate their own content. I hope they sue.
• Cyborgs: Obsolete Cyborg Dies in Power Outage
  58 years in an iron lung? Holy crap.

A complete stranger walks in, looks at me, and utters a mildly panicked “Hey man, can you give me a jump start?” He’s young, maybe 18 or 19.

“Problems?” I ask.

“Yeah, won’t start. And now the battery is dead.”

I walk with him out to the parking lot. The car is beautiful, some sort of 1970s Mustang, a deep shade of blue.

“Is this your car?”

“Well, sort of…”

“Sort of?” No way am I jumpstarting a stolen car.

“It’s my dad’s, I thought it’d be fun to take it out but now something’s wrong and he’s going to be pissed.”

“Ever seen Ferris Bueller’s Day Off?” I joke. He has. Not nearly as funny to him.

I hook him up to my Jeep and let the battery charge a little bit.

“Try it now.” He turns it over a couple of times, but nothing beyond sputtering and dying. Then it occurs to me what the problem is. He isn’t old enough to know how to drive a car where the driver is the engine management system. Sucks when you have to know how the machine works, because there isn’t a computer to think for you.

“Step on the gas pedal once, all the way down and up,” I suggest. Don’t know how he got the car here but right now it’s acting like it needs the choke set.

“You aren’t supposed to do that,” he says matter-of-factly.

“You aren’t supposed to steal your dad’s car, either, especially when you don’t know how it works. Trust me on this, this car isn’t fuel injected.”

This kid has no idea how to drive a carbureted vehicle. Guaranteed he’s never driven a rear-wheel drive vehicle in the snow or on a dirt road, either. Entire generations have now never had to deal with these technologies, for better or worse, because even their lawn mowers are fuel injected and have computers strapped to them.

Less than a minute later he was unhooked from my jumper cables and on his way.

I contend these kids are missing out, especially with the rear-wheel drive in the snow. Fun.

• Simulacra. | EUGENIZATION. A Personal Training Blog.
  “What if he’d applied the equivalent number of game-hours to playing a real guitar?” This is why I stopped playing “The Sims” many years ago. I realized I could go make real friends.

“It is impossible to defeat an ignorant man in argument.”

- William G. McAdoo

• Royal Pingdom » Smart and funny use of Google Adwords
  I like the ASCII art ones.
• ‘Miracle’ leaves that may help protect against liver damage | Science Blog
  Beer, beer, sea buckthorn, beer, beer, shot, beer…
• Office: What’s in Your Office Survival Kit?
  My office survival kit is a clean blank t-shirt, pair of socks, a pair of shorts, a stick of deodorant, a toothbruth and a travel-sized toothpaste, and a comb. And a supply of antacid and ibuprofen. All designed to give me options if I spill something.
• Licorice extract provides new treatment option for canker sores | Science Blog
  Hmm, I wonder if absinthe would work, then, too… I might have to try that.
• Former Prosecutor: ISP Content Filtering Might be a ‘Five Year Felony’ | Threat Level from Wired.com
  I’d like to see all of Charter in jail for wiretapping. But it also raises the question of the legality of things like IDS systems.
• jwz - was it my breath?
  It’s probably insensitive to call this an area-effect suicide… DOH!
• The Old New Thing : Microspeak: On-board (verb)
  The term “onboard” is the stupidest thing I’ve heard recently. Might as well say “waterboard.”
• Latest iPhone 2.0 Beta Adds Geo-Tagging to Photos - Mac Rumors
  I wish Apple would track these people down and nail them for NDA violations.
• jwz - words that annoy me, a continuing series
  Yes, when you say “email blast” I hear “spamming the crap out of unsuspecting people and betraying their trust.”
• 16% of US science teachers are creationists - being-human - 20 May 2008 - New Scientist
  I hope each of these 16% are found and prosecuted for violating the ban on teaching creationism. I continue to not understand why faith and facts have to conflict, except because someone says they do.

I ran into a case the other day where someone was reporting an operating system bug. A filesystem was 98% full, but an examination of that filesystem showed that it should only be 25% full.

It isn’t a bug. In order to understand why it isn’t, we need to know something about how files are stored, and then how they are deleted. A good place to start is the basic structure behind a UNIX-style filesystem, the inode. According to Wikipedia:

an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores basic information about a regular file, directory, or other file system object… Each file has an inode and is identified by an inode number (often referred to as an “i-number” or “ino”) in the file system where it resides.

Inodes store information on files such as user and group ownership, access mode (read, write, execute permissions) and type of file. There is a fixed number of inodes, which indicates the maximum number of files each file system can hold. Typically when a file system is created about 1% of it is devoted to inodes.

Very importantly, inodes only store file contents, not file names. Because file names are stored elsewhere an inode can have multiple names. Enter the hard link, which is a way to give the same file data multiple names inside a filesystem:

“A hard link is a reference, or pointer, to physical data on a storage volume. On most file systems, all named files are hard links. The name associated with the file is simply a label that refers the operating system to the actual data. As such, more than one name can be associated with the same data. Though called by different names, any changes made will affect the actual data, regardless of how the file is called at a later time. Hard links can only refer to data that exists on the same file system.”

On most operating systems a file is marked for deletion when the last name for it is removed from the filesystem:

The process of unlinking disassociates a name from the data on the volume without destroying the associated data. The data is still accessible as long as at least one link that points to it still exists. When the last link is removed, the space is considered free.

This is true for files that are not open. However, if a file is deleted but it is still held open by a process, the space doesn’t actually get marked as free until that process closes that filehandle.

That’s the “bug” — you can delete a file that is still open, but the space isn’t free. So a “du” might show 25% usage but a “df” shows 98%. This happens a lot with big log files. You go in, find the huge file, copy it somewhere, delete the original, and then note that nothing changed. The file isn’t there anymore but the space isn’t free. Lots of people scratch their head, declare it an OS bug, and reboot. A reboot fixes the problem, too, by globally closing every file, but had they restarted the process (or “kill -HUP” it, like syslog) it would have accomplished the same thing, by forcing the software to close and reopen the logs (and freeing the space).

This “bug” is actually a feature for some folks, though: it’s a way to securely use temporary files. A program could create a temporary file, open it, and then delete it so it isn’t visible in the filesystem, but it’s still there and usable to the program. In fact, the tmpfile() system call does this for you.

ONLamp has a great list of secure programming techniques as an excerpt from “Practical UNIX & Internet Security,” which mentions these topics and more. Also, if you aren’t familiar with inodes, directories, etc. those Wikipedia articles linked above are a good starting point. Consider it required reading if you’re a system administrator. :-)

• American Airlines adds fees for all checked bags - Travel News - Orbitz Travel Blog
  These people suck. Now I won’t be able to bring anything on board because everybody is going to bring their huge freaking bags. Idiots. Just raise airfares.

Robert Crawford has a post over at Server Specs about practical jokes for mainframe programmers. That got me thinking about all the stuff that I’ve been witness to:

• The ever-classic snapshot of my desktop as my desktop background. Yawn.
• Tape over the laser emitter on my mouse. Double yawn.
• Setting a user’s shell to a copy of Eliza.
• Setting a user’s shell to an emulator of another OS.
• Setting a user’s home directory to a floppy disk.
• Setting the shell prompt creatively, or very long, or very short.
• Setting the shell prompt with bell/^G characters in it. I currently don’t remember how this was done, probably echoing something into a .profile.
• Setting the resource limits for a user to very small values.
• Moving everything in my home directory to a hidden directory. Thankfully the prankster was good and left all my email in the right spots.
• Moving everything in my home directory to a hidden directory named exclusively with special characters (hyphens, etc.).
• Adding ‘logout’ to the end of a user’s .profile, .cshrc, etc.
• Sending mail to me from my own account. Boring but effective.
• Starting a process to slowly eat all the RAM on my desktop.
• Starting a process to slowly eat all the CPU on my desktop.
• Starting a process to slowly fill the process table on the desktop of the guy who did the last two to me.
• Removing my sudo rights to my own desktop and changing my root password.
• Adding cron entries to write things to my own terminals. Ingenious.
• Changing someone’s terminal settings. This is sort of boring, but I’ve always wanted to try setting someone’s terminal so that the key mappings are messed up. For example, set the ‘e’ key to backspace or something. I just haven’t had time or occasion to figure it out yet. :-)
• Changing a user’s shell to /bin/false. Boring.
• Changing a user’s shell to /usr/bin/perl. Much more effective when it’s actually a script which tells them they’re in Perl.

The best ones are ones that do absolutely no damage and require the victim to figure out an equally creative workaround. Or they have a very simple and possibly non-obvious workaround, like running a command via SSH to rename .profile.

:-)