Midnight is a Confusing Choice for Scheduling

Midnight is a poor choice for scheduling anything. Midnight belongs to tomorrow. It’s 0000 on the clock, which is the beginning of the next day. That’s not how humans think, though, because tomorrow is after we wake up! A great example is a statement like “proposals are due by midnight on April 15.” What you actually said: proposals aren’t welcome after April 14. What you probably meant: you want the proposals before the date is April 16. There’s a 24 hour difference there, and if you enforce the deadline accurately people are going to complain because they were all thinking the second thing (before April 16). Similarly, this is a problem in change notices and customer communications. When you say …

Read More

Standards, to and with Resolve

As the holiday season has progressed I’ve spent a bunch of time in the car, traveling three hours at a crack to see friends and family in various parts of Midwestern USA. Much of that travel has been alone, my family having decided to ensconce themselves with my in-laws for the full duration of the week. That has left me ample time to sing aloud in the car, take unplanned detours to collect growlers of beer from esteemed breweries, and to think. I don’t do New Year’s resolutions. I’m not against them, per se, but I just think they’re too conveniently abandoned. I like the noun form of “resolve” better — a firm determination to do something. I aspire to …

Read More

Big Trouble in Little Changes

I was making a few changes today when I ran across this snippet of code. It bothers me. /bin/mkdir /var/lib/docker /bin/mount /dev/Volume00/docker_lv /var/lib/docker echo “/dev/Volume00/docker_lv /var/lib/docker ext4 defaults 1 2″ >> /etc/fstab “Why does it bother you, Bob?” you might ask. “They’re just mounting a filesystem.” My problem is that any change that affects booting is high risk, because fixing startup problems is a real pain. And until the system reboots the person who executes this won’t know that it works. If it doesn’t work it’ll stop during the boot, sitting there waiting for someone with a root password to come fix it. So you’ll have to get a console on the machine and dig up the root password. Then …

Read More

CentOS 7 Refusing VMware vSphere Guest OS Customizations

So I just spent two hours of my life trying to get my CentOS 7 VM template to deploy correctly with a vSphere customization specification. No matter what I did it would customize the VM, then uncustomize it, essentially leaving me with the template again. I finally asked our oracle and savior, Google, and two amazing things occurred. First, I found the answer. About three weeks ago a fellow named Jeff Burns asked this same question on Server Fault, then answered his own question five minutes later (this is often what happens to me immediately upon filing a support case). He built on something I’d seen in /var/log/vmware-imc/toolsDeployPkg.log, where the VMware Tools couldn’t figure out what the OS is and …

Read More

Tech Bloggers: Punctuation Goes Inside Quotation Marks

One of the biggest differences between writing code for machines and writing English-language text for humans in the United States is the use of quotation marks. When you’re programming a computer a set of double quotation marks indicates a string, which is an atomic entity. As such, punctuation goes outside the quotes to delimit lists and whatnot. #include <stdlib.h> string animals[4] = {“Goat”,”Sheep”,”Cow”,”Platypus”}; This is not how it works when you’re writing in the English language. Periods and commas always go inside the double quotation marks in English. Incorrectly punctuated sarcasm: We all know how that piece of software “works”. Correctly punctuated sarcasm: We all know how that piece of software “works.” Incorrect: Her number is “867-5309”. Correct: Her number …

Read More

Keeping My Blade Options Open

One of the types of advice I really appreciate is that which helps me to keep my options open. I have a team from Dell in the office this week, configuring a giant pile of equipment we bought. The equipment includes a bunch of blade servers. We’ve relied on rack-mount equipment for decades, but with a push towards a private cloud we opted to jump into the early 21st century with blades. I’ve had relatively little experience with blades so it’s nice to have more experienced people around. When I’m designing a system I always try to figure out what it’ll need to look like four years from now. Seeing the future is the hardest part of designing a system. …

Read More

Better Linux Disk Caching & Performance with vm.dirty_ratio & vm.dirty_background_ratio

This is post #16 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” In previous posts on vm.swappiness and using RAM disks we talked about how the memory on a Linux guest is used for the OS itself (the kernel, buffers, etc.), applications, and also for file cache. File caching is an important performance improvement, and read caching is a clear win in most cases, balanced against applications using the RAM directly. Write caching is trickier. The Linux kernel stages disk writes into cache, and over time asynchronously flushes them to disk. This has a nice effect of speeding disk I/O but it is risky. When data isn’t …

Read More

Account for the Bandwidth-Delay Product with Larger Network Buffers

This is post #14 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” At times we can refer to certain network concepts with a “pipe” analogy, where bandwidth is the diameter of the pipe and latency is the length of the pipe. A garden hose has a certain limited amount of “bandwidth” because it has a small interior diameter, and has a lot of latency because it is so long. Water you put in one end of the hose takes a while to come out the other end. A house in the USA likely has a main sewer pipe with 4 inches of “bandwidth” so it can move more, …

Read More

Use a RAM Disk to Improve Disk Access Times

This is post #15 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” One of the biggest things folks in IT worry about is data loss. We go to enormous lengths to protect our data, using backups, snapshots, remote replication, rsync, scp, temporary copies in our own home directories… you name it. The thing is, as we look at our systems we sometimes discover that our applications do a lot of writing of temporary files. These temporary files often don’t need any particular protection because they’re transient, yet we write them to our expensive, already overtaxed disk arrays, commit the writes over long distances to our DR sites, …

Read More

Use Paravirtualized Devices On Your Virtual Machines

This is post #13 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” para- a prefix appearing in loanwords from Greek, most often attached to verbs and verbal derivatives, with the meanings “at or to one side of, beside, side by side” ( parabola; paragraph; parallel; paralysis ),”beyond, past, by” ( paradox; paragogue ); by extension from these senses, this prefix came todesignate objects or activities auxiliary to r derivative of that denoted by the base word ( parody;paronomasia ), and hence abnormal or defective ( paranoia ), a sense now common in modernscientific coinages ( parageusia; paralexia ). Paravirtual drivers are ones where the virtualization platform does …

Read More