Let’s Just Keep An Eye On The Time

Double Quote“You’re asking me how a watch works. For now, let’s just keep an eye on the time.” – Alejandro, Sicario

I’ve enjoyed the eclectic roles Benicio del Toro has been playing these last few years. His appearance in recent space movies reminded me of this quote of his from the movie Sicario. Often enough in our own technological roles we are asked to explain ourselves, explain why something is the way it is or why we want it to be a particular way. How do you convey to someone in just a minute the years of school, decades of experience, days in noisy data centers, nights bringing systems back online, hours staring at configurations that are wrong and scripts that don’t work, dumb mistakes, clever moves, seemingly clever moves that were really dumb mistakes… all that led to this particular moment, this particular system design, this particular decision?

I kinda just want to tell them to keep an eye on the time.

Fixing Veeam “Can’t Delete Replica When It Is Being Processed” Errors

Floppy DiskI’ve used Veeam Backup & Replication for a long time now, and when we restructure storage, redeploy VMs, or change our replication jobs we sometimes get situations where we get the error:

Error: Can't delete replica when it is being processed

Here’s how I fix it. As always, your mileage may vary, and free advice is often worth what you paid, especially from a stranger on the Internet. Veeam support is probably a safe but much higher latency source of non-free advice.

  1. Stop the affected jobs and disable them.
  2. Ensure that the replicas are gone, from both the VMware environment (vCenter) and in Backup & Replication (Replicas -> Ready, then right-click and Delete From Disk). Don’t delete it from the configuration or you’ll have to go in and edit the VM Exclusions.
  3. Browse the target datastore and remove any orphaned disk files or folders associated with that replica.
  4. Clone the affected job (by doing this you make sure that the replica you’re about to create has nearly identical properties to what the other job would have created). Change the virtual machine list to be only the affected VM or VMs.
  5. Run the replication job. Let it complete. It should create a replica of the VM.
  6. Edit the original job. Enable “Low connection bandwidth (enable replica seeding).” In the “Seeding” tab check “Map replicas to existing VMs” and click the “Detect” button to automatically map most/all of them. Check to make sure the affected VMs are detected, and fix it if not.
  7. Run the original job. It should complete without error now.
  8. Edit the original job and disable “Low connection bandwidth (enable replica seeding).” I’d run the job again just to make sure everything is good.

Good luck.

7 Ways IT Staff Can Prepare for the Holidays

Calendar IconFor us IT types it is important to maintain a good balance between work and our lives. Just as they say that good fences make good neighbors, I’ve found that a good delineation between work and home improves both. The holiday season is taxing, though. People rush around trying to wrap up loose ends, they’re using vacation they’re going to lose, and they’re generally scattered and distracted, which isn’t a good thing.

If you’re lucky enough to work somewhere with a true 24×7 operations center then coverage over the holidays is already thought out. However, most IT staff in the world aren’t in places like that. Here are some thoughts I have about how to defend your time off over the holiday season.

1. Talk to your manager about expectations.

Ideally your manager has told you what they expect over the holidays, but if that isn’t the case you might have to bring it up. Certainly there is a range of competency when it comes to leadership but hopefully you can have an honest & fair discussion about staff availability.

I always have a million questions about expectations. For example, will anybody be working during the holidays? Remember that not everyone follows a Christian religion. What I think of as Christmas will just be “Monday” to several of my coworkers. Will they need support? Does that mean you’re on call? If you’re on call, what are the expectations? Can you support them remotely, or might you need to go into the office? What timeframe do they expect you to respond in? What timeframe do they expect you to get to the office in? How will they contact you?

If your boss doesn’t have answers seize the chance to propose something reasonable. I might start with “no calls on the actual holidays, and one person will be available by phone, but not on site, during business hours between Christmas and New Year’s Day” and go from there.

Remember that this stuff is work, so any work you do reduces any vacation time you’re taking. Keep that in mind if you’re trying to use up vacation.

2. Publish & honor holiday hours.

Many problems in IT are communication issues. Getting called on your day off is both a communication issue and a problem. Head it off by making a calendar of who’s available when so your team knows who is in and who isn’t, and who’s on call. Make that a PDF and send it to people so it’s in their email. That way they can get to it if they’re traveling.

The second part of that is honoring those hours. Where I work you’re either working or you’re not working. If you’re working you’re fair game, but if you aren’t working we don’t call. If we do call there’s no expectation that you’ll answer.

My teammates will text the group if there’s something going on, which is nicely non-disruptive. If we’re busy we ignore it, if we have a few minutes we’ll jump in, or suggest a course of action. Above all, respect your team’s personal lives and time away from work.

3. Institute a pre-holiday change freeze.

There have been several studies done over the last couple decades that showed a high percentage (85% or more) of incidents & outages are caused by IT staff themselves. To me that means if we’re all on vacation there’s a much smaller chance something bad will happen!

You don’t want to deal with preventable problems when you’re trying to leave on vacation. You also don’t want to have to call or be called to help someone fix something they could have left until January. To avoid that, invoke a change freeze. For example, you could declare that no changes will be made between December 18 and January 2 unless they are to resolve a critical issue. As Shakespeare would put it, “on pain of death.”

Every IT shop I’ve ever worked in has had lots of stuff to do that didn’t involve changing production systems. Find something else to do. Write some documentation. Build a wiki for your documents. Vacuum your office. Vacuum the data center. Clean out that closet of old cables. Demo some new software. Restock printer supplies. Brush up on your Linux admin skills. Learn a scripting language. Test your backup system by restoring something. Take long lunches and do team building exercises at a pub. You’ll figure it out. One last thing, though — all these things are part of your job. Stand your ground if someone says you aren’t working, because they’re wrong.

4. Find opportunities for users to help themselves.

One thing you can do is find opportunities for your users to help themselves. Write a “what to do if your IT person is out” document. There’s no reason that a user couldn’t reload paper in a printer, or reboot their PC, or do some basic troubleshooting before they interrupt your vacation time..

For example, I once found that adding a second printer to people’s PCs, a printer in a neighboring department, for instance, got me out of 95% of urgent printer calls. People get really angry when they can’t print right before a meeting. Write up some tips and give them permission to help themselves out.

5. Socialize limited availability and light work schedules.

Now’s the time to mention at meetings that the holiday week will be lightly staffed, so turnaround times might be longer, project work will be on hold, and issues that aren’t absolutely life or death will be tabled. I’ve always found that by mentioning these issues early you get people thinking about it in a non-threatening way, and scheduling assumptions can be sorted out proactively & calmly.

6. Set up an on-call phone number.

It’s one thing for your team to know your personal cell phone number, but you really don’t want everybody in your organization knowing it. Trust me. If you are doing an on-call rotation that involves actual calling you should find a way to forward a business phone number to whoever is responding. Maybe it’s tricks with Google Voice, or someone logs into Burner or Signal or something when it’s their turn. Be creative but make sure it’s reliable. Make sure to document it and send the document to people via email so they can find it when they need to.

7. Set up & test remote access methods.

The time to test your ability to get in remotely is BEFORE you need it. Make sure you can get in from the road if you’re traveling. Do you need a hotspot? Can you VPN in? What happens if the VPN concentrator can’t talk to your Active Directory? Can you get to the consoles of all of your devices? Does everyone on your team know how to get to everything?

Anything you can’t reach remotely is potentially a reason you’ll have to go into the office. Going into the office means having to put on pants, comb your hair, log out of Overwatch, sober up — all things we just don’t want to have to do while we’re on vacation. So get it together, folks, and lay the foundation for a very merry end of December.

Good luck!

Consistency Is the Hobgoblin of Little Minds

Ever heard someone tell you “consistency is the hobgoblin of little minds?”

They’re misquoting Ralph Waldo Emerson by leaving out an important word: foolish.

That’s like leaving out the word “not” in a statement. The whole meaning changes because of the omission. We can all agree that “I am on fire” and “I am not on fire” are two very different statements. The same is true here. Let’s examine the actual quote:

A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall.

As with most things, context matters, which is what makes this quote inappropriate almost everywhere I see it used. In the context of IT, with consistency, a great soul can trade meaningless & soul-crushing work for important & strategic tasks, moving their organization forward rather than struggling just to keep up. In IT, consistency is generally a good thing, and when it is delivered via standards and automation it forms a stable & solid & predictable foundation on which we can build towering pinnacles of applications and services. Stability and predictability are important things to app developers, end users, and those of us that want to take a vacation from time to time.

However, there are foolish consistencies. When your automation becomes handcuffs and not an enabler it’s foolish. When your standards are held so tightly that you cannot enable new business ventures because of them it’s foolish. When efficiencies aren’t taken advantage of, new technology and methods eschewed, and/or positive changes avoided with the excuse of “standards” or “that isn’t the way we do things” it’s foolish.

Neither standards nor our tools are an end unto themselves. They exist to enable greater things, and when that stops being true we need to change them so they are helpful to us again. After all, having two standards is still better than having 1500 one-offs.


Advice On Downgrading Adobe Flash

VMware has a KB article out (linked below) about the Adobe Flash crashes that happen if you’re running the latest version of Flash ( A lot of us were caught off guard recently when our PCs updated themselves and we couldn’t get into our VMware vSphere environments.

The VMware KB article suggests downgrading your Flash client. Left by itself this is completely irresponsible advice.

1. The Adobe Flash update addresses a critical security vulnerability that is being exploited in the wild. The security advisory (linked below) states:

Adobe has released a security update for Adobe Flash Player for Windows, Macintosh, Linux and Chrome OS. This update addresses a critical type confusion vulnerability that could lead to code execution.

Adobe is aware of a report that an exploit for CVE-2017-11292 exists in the wild, and is being used in limited, targeted attacks against users running Windows.

(as an aside, Adobe acknowledges Kaspersky Labs staff, which makes me think that they’re making good on their promises to figure out how Russian hackers used their software to exfiltrate NSA data).

2. If you downgrade your Flash installations you will need to disable the auto-updaters, which is what got us all into these situations. I don’t know about you but I always forget to re-enable the updaters, and that’s bad.

3. There are workarounds. The HTML5 client, though incomplete, gets many people back in business. Microsoft Edge and Internet Explorer seem to work with Flash on Windows 10 1703, too, at least for all my team’s environments.

So what’s my advice?

  • Limp along with Microsoft Edge and the HTML5 client until VMware updates their clients. I think it’s safe to assume they’re working on it. Start making plans to patch your vCenter in the next few weeks.
  • If you don’t have the HTML5 client you can get it as a VMware Fling (link below).
  • If you absolutely have to downgrade Flash don’t run the vulnerable Flash on a PC you use for anything else. It’s annoying but you can survive a few weeks of this, provided you’re running a supported version of vSphere.
    1. Use network- & host-based firewalls to prevent all traffic that isn’t destined for your vSphere implementations. You’ll probably need to allow DNS, as well, but I’d keep it really locked down. I would even think twice about joining it to Active Directory.
    2. You should already be running antivirus and antimalware on your systems but it’s especially important for systems that are intentionally out of date.
    3. Use a virtual machine running in VMware Workstation for the insecure client. Make it non-persistent and use it for nothing else. Or a Windows Server installation with Terminal Services enabled.
    4. Put a calendar reminder in for your team to clean this whole thing up in a month.
    5. If you have dedicated IT security personnel (CISOs and such) reach out to them proactively. Make a business case around this — you need to do this to be able to support the environments, but you’re being responsible about the risk.
  • If you’re running an unsupported version of vSphere you need to upgrade ASAP. This is a great business driver for it. Never let a crisis go to waste! vSphere 5.5 goes end-of-support on 09/19/2018 so I’d even consider using this as a driver to get to 6.5…

Good luck & stay safe.


Stop Chrome Autoplay

If you didn’t catch this on Twitter:

In short, go to chrome://flags/-policy and set it to “Document user activation required.”

It’s funny how simple things can be so virally popular.

While Chrome can sync settings between browsers where I am logged in, I have got to figure out if there’s an API to set Chrome configuration options automatically…

Software is Always Broken

Life PreserverI’m sitting here watching my iPhone update to iOS 11.0.1. Apple says that there are just a couple of fixes: some security updates and a fix for the Exchange email problems.

The update is sure taking a while, though. That’s consistent with my knowledge of how software development works. Color me skeptical that the first point release of a new iOS only has a couple of changes. My bet is that there are hundreds of fixes for all sorts of problems reported during the beta, but weren’t large enough to stop the release.

Development of software like Apple’s iOS or VMware’s vCenter never stops. At a certain point someone takes a snapshot of the way it looks and decides that it acts correctly enough to ship. Out the door it goes, called 11.0.0, or 6.5.0, or whatever.

Behind the scenes development never stops, though. People keep reporting bugs and the development groups keep fixing them. Most of the problems they fix will never be known to the end users. If you had a bug open with Apple or VMware you might find out that it’s fixed in a particular release, but without that particular inside knowledge you’d never have known.

What you do see is build numbers and version numbers, though, assigned to the snapshots that became releases.. Read the notes for a point release of VMware vCenter and there will be a handful of problems resolved and one or two new features added. The build number of vCenter will jump by 387,000, though. Nearly 400K builds for a few changes? Yeah, right. In this particular case (vCenter 6.5.0 build 5318154 to 5705665) that’s around 6500 builds a day. Something else is clearly going on.

To me, these are the ripples on the surface that belie the chaos underneath. It’s nice to think of products like iOS or vSphere as a glassy lake our IT boats can glide across, but it is more productive for us to remember that under the surface are the software equivalents of fallen trees, sandbars, litter, and dead fish. The people with the nets pulling the dead fish out sometimes fall in, too. Sometimes we get new software bugs in the fixes for other problems.

That’s why I always laugh at the people who say “if it ain’t broke don’t fix it.” Truth is that it’s all broken and we’ll never know just how bad the situation really is. The best thing we can do is move forward. Keep current on our patching, stay responsibly close to the current major version. Build and use test environments. Grow into new initiatives and deployments. Test our backups and DR.

And wear personal floatation devices for when our boat hits a rock.



Calibrate Your Monitor

When I build a new computer one of the things I do as part of the setup is calibrate the color of the monitors. It’s actually pretty amazing how much better things look after just a few minutes of adjustments. It’s also nice to have the monitors synchronized, so if I move a window between them it doesn’t change color.

I use Microsoft Windows 10 (1703 as of this writing) on all my desktops now, and here’s my process. Apple appears to have a similar calibration tool built in to MacOS, and all my Linux boxes are headless, so you are on your own. Sorry.

1. Reset the monitor(s) to factory settings. On my Dell LCD panels there’s a “Reset Display Settings” and a “Factory Reset.” Go for Factory Reset.

2. Set the monitor energy mode to the one that doesn’t have automatic dimming. On the Dell panels it’s called “Standard.” Automatic dimming is nice on a laptop, tablet, or phone but not on the desktop. If you’re worried about power consumption use the system power settings to put the display to sleep when it’s idle.

3. In Device Manager, make sure that your monitor has the best driver. Right-click the monitor, choose “Update Driver” and tell it to search automatically. Alternately you can download the driver from the manufacturer, but manufacturers don’t really focus on keeping monitor drivers updated. For example, Dell has a Windows Vista-era driver for the monitors I’m looking at now, and it comes with an ICC color correction profile. Your mileage may vary.

Windows Device Manager - Calibrate Your Monitor

4. Do the same for your video cards. I generally prefer the MS drivers because they’re automatically updated with the OS, and don’t need the heaps of extra spyware crap NVIDIA and AMD install on your PC. I also don’t game a lot, either, so YMMV. Anyhow, a driver change tends to screw up the calibrations, so now’s the right time for a change if you’re going to make one.

5. Disable Night Light (Settings -> System -> Display -> Color -> Night light). A huge shout out to the f.lux folks that started this all).

6. Go to “Display adapter properties” at the bottom of the Display settings. Click the Color Management Tab, and click the “Color Management”button.

Display Adapter Properties - Calibrate Your Monitor

7. Click the “Advanced” tab, then the “Calibrate Display” button about 3/4ths of the way down. This is where you will get to actually calibrate your monitor. If you installed a monitor driver that came with a ICC profile you might want to try setting it in the “Device Profile” dropdown, at least to see if it helps.

Color Managaement Advanced - Calibrate Your Monitor

8. Follow the instructions precisely, and repeat for every monitor you have. It might offer to start the ClearType tuner. It’s a good idea.

9. When you’re done turn Night Light back on, or install f.lux. In either case (and on my tablets and phones) I adjust the color temperature back up a little from the deeper reddish hues it sets by default.

10. If your monitor supports it set the “Menu Lock” set it. This will reduce wear & tear on your tasers and stun guns, which I consider an appropriate response for anyone who tampers with one’s monitor settings, or touches someone’s screens. You might want to look up how to unlock it first (usually just holding the menu button for 15 seconds or so).

As always, good luck.

Let’s Prosecute Unlicensed Engineering in IT

Have you been watching this whole dustup with the Equifax CISO, and how people are saying that she is unqualified because, instead of a Computer Science degree, she had an MFA in music composition? Not surprisingly, there’s a massive backlash from the IT community, much of which doesn’t have a computer science degree, either. That’s part of the appeal of technology for many — on the Internet nobody knows you’re a dog. I’m a mutt, too. I’ve always found computer science programs intentionally inaccessible, with the faculty actively eschewing any form of practical curricula because they’re not a technical college. Snobbish? Yeah. Not my style.

What I find very interesting in all of this is the ignorance of some of the folks throwing rocks in this debate (surprising, I know). I’m particularly interested in a word that many have in their titles: engineer.

I live in the United States, in the state of Wisconsin, and as with many other states we actually have law that sets a legal definition of engineering:

WI 443.01 (6) “Practice of professional engineering” includes any professional service requiring the application of engineering principles and data, in which the public welfare or the safeguarding of life, health or property is concerned and involved, such as consultation, investigation, evaluation, planning, design, or responsible supervision of construction, alteration, or operation, in connection with any public or private utilities, structures, projects, bridges, plants and buildings, machines, equipment, processes and works. A person offers to practice professional engineering if the person by verbal claim, sign, advertisement, letterhead, card or in any other way represents himself or herself to be a professional engineer; or who through the use of some other title implies that he or she is a professional engineer; or who holds himself or herself out as able to practice professional engineering.

Does the work of an information technology professionals affect the public welfare? I think the Equifax case is example enough: yes. Do IT professionals safeguard life, health, or property? Absolutely. In fact, information technology touches most of what is defined in this statute.

It gets more interesting in 443.02 (02):

WI 443.02 (02) No person may practice architecture, landscape architecture, or professional engineering in this state unless the person has been duly registered.

443.04 and 443.09 cover how you become registered. In short, you need an engineering degree from a school accredited in such topics, and you need to pass a written or written & oral examination.

So what happens if you don’t do this, and call yourself a professional engineer anyhow?

443.18 Penalties; law enforcement.
(1) Unauthorized practice; penalty.
(a) Any person who practices or offers to practice architecture, landscape architecture, or professional engineering in this state, or who uses the term “architect,” “landscape architect,” or “professional engineer” as part of the person’s business name or title… may be fined not less than $100 nor more than $500 or imprisoned for not more than 3 months or both.

A man in Oregon was recently fined $500 for doing just this: claiming to be an engineer in a public forum despite not being licensed to practice in Oregon. Specious? In that case, maybe. But if we’re going to throw rocks over CS degrees let’s get serious and enforce the actual laws around engineering, too. Especially if you are designing, building, and/or operating systems that affect others. If one type of engineer isn’t allowed to build an unsafe bridge, skyscraper, or canal, has a mandatory code of ethics, and has to go to hell and back to be licensed why do others with none of this training or effort get to use the title while they build & operate unsafe and unethical systems?

Frankly, it’d be easy to find offending folks. Start by dredging job boards and looking for postings that have names like “Site Reliability Engineer” or “Network Engineer” or “Systems Engineer” or “Software Engineer.” I bet even Equifax has some “engineers.” From there, it’s a subpoena or two and a $500-a-pop form letter. Maybe even some injunctions against organizations, too, for their use and tolerance of unlicensed engineers.

A small price to pay to ensure unqualified people aren’t endangering the public welfare. Right?

Help Me Name a Team, Win $50

I know all you nerds have a few minutes to help me crowd source the solution to a problem. I’m also willing to offer $50 in the form of an Amazon gift card to someone with the right solution.

We want to rename our group at work. We are currently “Systems Engineering” (abbreviated “SE”). There are a number of decent reasons why we want a different name, and I’m not going to get into that. Let’s just suffice it to say that the change is definitely going to happen in the next week or so.

My group designs, builds, and operates compute & storage infrastructure, for all sizes of customer, and all manner of security level (basic all the way to NIST 800-53 “High”). We provide traditional system administration services. We do VMware, Linux, Windows, Solaris, AIX, Mac OS, everything. We also install & maintain applications for customers, and specialize in automation technologies (Puppet, etc.). We often serve as the people who bridge gaps between other groups, and between vendors and the organization, and are often the glue that holds a solution together.

We don’t do anything but the basics of databases or networking, other groups hold the deep expertise there. Nor do we manage facilities. There is also a separate CISO-led security group so they provide a lot of the policy that drives our designs and operations. And other, large, enterprise-y applications (ERPs, email & calendaring, identity management, etc.) have their own support teams.

Another group formed where I work and calls themselves “CAOS” — pronounced as chaos — and we’re a little jealous of that. We’d like a clever name, too, especially one that sounds like supervillians. So far the best name put forth is simply just “Systems & Application Engineering” or “SAE.” It isn’t bad but I think there’s better out there. At least with SAE we can make viscosity jokes about slow projects for the next decade.

My question to you is: can you come up with a clever, HR-safe acronym & name for this team that reflects what we do?

Put your suggestions in the comments. Quality is good but volume works, too — others might riff on your ideas.

If we pick your suggestion I’ll personally send you a $50 gift card to Amazon (this is me doing this, not my employer). If you aren’t in the USA I’ll see if I can do it in the currency of your choice. If a suggestion is made twice the first one wins it. If no suggestions are chosen I will put all the suggestions in a pool and draw one randomly for the prize, as a thank you for participating. Please make sure your email is correct in the comments!

Ready… go!

%d bloggers like this: