Fixing Veeam Backup & Replication Proxy Install Errors

Every once in a while I struggle a little to add a new Veeam Backup & Replication hot-add proxy. If you’re like me and seeing proxy install errors maybe some of these will fix you up. This is what worked for me on Windows Server 2016 when I was getting error 0x00000057, “Failed to create persistent connection to ADMIN$” and some other unhelpful messages. If you’re using a hardened Windows installation all bets are off, since the goal of hardening is to intentionally disrupt remote access. I’d get it running with as close to a stock Windows installation as possible and then work from there if you need to secure things further. There are also ways to manually install the Veeam …

Read More

vSphere 6.7 Will Not Run In My Lab: A Parable

“Hey Bob, I tried installing vSphere 6.7 on my lab servers and it doesn’t work right. You tried using it yet? Been beating my head against a wall here.” “Yeah, I really like it. A lot. Like, resisting the urge to be irresponsible and upgrade everything. What are your lab servers?” I knew what he was going to say before he said it. “Dell PowerEdge R610s.” I was actually surprised it was that new, and rack-mountable. “Yeah, you’re out of luck. CPUs before the E3/E5/E7 family didn’t have VT-x extensions in them to make virtualization easy so VMware had to do this thing called binary translation. vSphere 6.5 was the last release that they supported that on because, frankly, it’s slow …

Read More

Should We Panic About the KPTI/KAISER Intel CPU Design Flaw?

As a followup to yesterday’s post, I’ve been asked: should we panic about the KPTI/KAISER/F*CKWIT Intel CPU design flaw? My answer was: it depends on a lot of unknowns. There are NDAs around a lot of the fixes so it’s hard to know the scope and effect. We also don’t know how much this will affect particular workloads. The folks over at Sophos have a nice writeup today about the actual problem (link below) but in short, the fix will reduce the effectiveness of the CPU’s speculative execution and on-die caches, forcing it to go out to main memory more. Main memory (what we call RAM) is 20x slower than the CPU’s L2 cache (look below for a good link showing …

Read More

Software is Always Broken

I’m sitting here watching my iPhone update to iOS 11.0.1. Apple says that there are just a couple of fixes: some security updates and a fix for the Exchange email problems. The update is sure taking a while, though. That’s consistent with my knowledge of how software development works. Color me skeptical that the first point release of a new iOS only has a couple of changes. My bet is that there are hundreds of fixes for all sorts of problems reported during the beta, but weren’t large enough to stop the release. Development of software like Apple’s iOS or VMware’s vCenter never stops. At a certain point someone takes a snapshot of the way it looks and decides that …

Read More

Install the vCenter Server Appliance (VCSA) Without Ephemeral Port Groups

Trying to install VMware vCenter in appliance/VCSA form straight to a new ESXi host? Having a problem where it isn’t listing any networks, and it’s telling you that “Non-ephemeral distributed virtual port groups are not supported” in the little informational bubble next to it? Thinking this is Chicken & Egg 101, because you can’t have an ephemeral port group without a Distributed vSwitch, and you can’t have a dvSwitch without a vCenter, so how do you install vCenter when you need something that only vCenter can create? Yeah, me too. Here’s the secret, though: don’t remove the default “VM Network” port group, or if you did, put it back, and restart the installer (or just back up to select the host …

Read More

Why Use SD Cards For VMware ESXi?

I’ve had four interactions now regarding my post on replacing a failed SD card in one of my servers. They’ve ranged from inquisitive: @plankers why would you use an SD card in a server. I’m not a sys admin, but just curious. — Allan Çelik (@Allan_Celik) January 22, 2015 to downright rude: “SD cards are NOT reliable and you are putting youre [sic^2] infrastructure at risk. Id [sic] think a person like you would know to use autodeploy.” Aside from that fellow’s malfunctioning apostrophe, he has a good, if blunt, point. SD cards aren’t all that reliable, and there are other technologies to get a hypervisor like ESXi on a host. So why use SD cards? 1. Cost. Looking at …

Read More

You Cannot Use open-vm-tools to Customize VMs

Homer Simpson: Kids: there’s three ways to do things; the right way, the wrong way and the Max Power way! Bart: Isn’t that the wrong way? Homer Simpson: Yeah, but faster! My biggest pet peeve with open source is that projects don’t ever solve whole problems. They get 60% of the way to solving a whole problem and then run off to chase another squirrel. The most recent example of this is VMware’s recommendation to use the open-vm-tools packages that ship with modern distributions of Linux. Dumbest recommendation ever. Why? Because the project got to 60% of the solution and stopped, effectively solving no problems for anybody. From what appears to be a VMware employee on the open-vm-tools mailing list archives: > …

Read More

CentOS 7 Refusing VMware vSphere Guest OS Customizations

So I just spent two hours of my life trying to get my CentOS 7 VM template to deploy correctly with a vSphere customization specification. No matter what I did it would customize the VM, then uncustomize it, essentially leaving me with the template again. I finally asked our oracle and savior, Google, and two amazing things occurred. First, I found the answer. About three weeks ago a fellow named Jeff Burns asked this same question on Server Fault, then answered his own question five minutes later (this is often what happens to me immediately upon filing a support case). He built on something I’d seen in /var/log/vmware-imc/toolsDeployPkg.log, where the VMware Tools couldn’t figure out what the OS is and …

Read More

Update to VMware vCenter Server Appliance & NTP Issues

Earlier today I posted “VMware vCenter Server Appliance 5.5.0 Has An Insecure NTP Server.” One of the reasons I like VMware is that they’re responsive to customer issues. This situation is no different. I just spoke with a few guys involved in VMware security, and this is what I’ve learned. 1. There has been mitigation information available internally to VMware Support/GSS since shortly after the vulnerability was published. If you call VMware Support your best bet is to reference the CVE number, CVE-2013-5211. I have not called VMware Support to confirm this, or to verify that they’re able to properly resolve the issue if you don’t reference the CVE number. In the future I’ll make sure to reference the CVE number if …

Read More

VMware vCenter Server Appliance 5.5.0 Has An Insecure NTP Server

Update: I have updated this article to reflect some new information provided by VMware. I have also published new notes and discussion as a separate blog post. On January 10, 2014 a vulnerability in ntpd, the Network Time Protocol daemon, was made public (US CERT VU#348126): UDP protocols such as NTP can be abused to amplify denial-of-service attack traffic. Servers running the network time protocol (NTP) based on implementations of ntpd prior to version 4.2.7p26 that use the default unrestricted query configuration are susceptible to a reflected denial-of-service (DRDoS) attack. Other proprietary NTP implementations may also be affected. I have encountered several vCenter Server Appliances, version 5.5.0 build 1476327 and older, that were exposed to the general Internet, and have …

Read More