Retrieve an SSL Certificate from a Server With OpenSSL

Security Shield

I was setting up VMware vRealize Automation’s Active Directory connections the other day and I needed the public SSL certificate for the AD DCs to authenticate correctly. You can use OpenSSL to get that information.

I used a Linux shell but this should be do-able from a Mac or with OpenSSL installed on Windows, too.

If you wanted to read the SSL certificates off this blog you could issue the following command, all on one line:

openssl s_client -showcerts -servername lonesysadmin.net -connect lonesysadmin.net:443 < /dev/null

In this case you’ll get a whole bunch of stuff back:

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = lonesysadmin.net
verify return:1
Certificate chain
0 s:/CN=lonesysadmin.net
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFWDCCBECgAwIBAgISA/05rMV6H+0LKP7uo3EE2F5zMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODExMTIwNTEzMTBaFw0x
OTAyMTAwNTEzMTBaMBsxGTAXBgNVBAMTEGxvbmVzeXNhZG1pbi5uZXQwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC22CK5EPHggy7q6qgqiYObGumi4b6X
DV/xKXauS5P2w2zjUFnHO01KRPDQ2owrc4opNRbngqanBI6llWTlBTrMCJSCa8sQ
uE6PEp7Hs94y1xHKKWQ+Lkk8ha99E50plAAU0CY3M7qqJ5Js5Q6L+MZ94H7/4Vvr
c0Ojiu6iXug5YmeNteqE3gzJvQcNoEk8js4HUUuH1FJNm1dLtKFY1/NCut6M39zC
1QOh7YfuGj/DgZD0le24SlBFYgv/2kDwVdPesIktyuV0aPZ+gihLFAjQotCvfhZ1
lQ8/+n9gFlG0vpjVdv8ZurixeyUykVk/Xq9HZLPwaDCubJ2NT+tZoqyXAgMBAAGj
ggJlMIICYTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFMnifCC04NozOYpDuRzN16P
+WuOMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEB
BGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZy8wGwYDVR0RBBQwEoIQbG9uZXN5c2FkbWluLm5ldDBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AHR+2oMx
rTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQe8xWAAABZwaNaRwAAAQDAEgwRgIhAOWY
Xz2ucXa1Src4Ec6cBR6C6TNttfkGGR9mzTGM/SvZAiEA+EgD4cX9Wl+fqctk3iB8
ho/efgo3Urh9BPPQYefb4zQAdQApPFGWVMg5ZbqqUPxYB9S3b79Yeily3KTDDPTl
RUf0eAAAAWcGjWrZAAAEAwBGMEQCIGmIhmJ1x68Z0s32ZTTv6NBD1pzRaSTo92t8
mybQWsnsAiAg+R3O4ruTA/ao1OZfjlg5GG8/+9Z6qtTGLF0bK+cyDTANBgkqhkiG
9w0BAQsFAAOCAQEASsi4FWVLjkLG04BgtnPWyxWoPTFY16MDTLpat6Clf9oQTjb4
KcpNn9tYlLyv9NWNYnQiD351IvDAoPG1Lk4iGdQlo1kFNlrjyozZlQGax7g1XuXX
9OasSVK5aJtIGStR7J2NQsgbLgbY9AajC9BWE7lCPE9AsRIDj96nN3DgbL8hng3d
2G2CBUEYC9+FqnhpqNUXuogLNr9SUm6AODLsRMGoQ0lXwLmjrb++tKuEn55SlHHF
GvDHKB/qCfjLFpByk05J49v2qDBJcxmOkyTHyi9TnDBplWDRCHO99J6qgMorWma3
UavqxAHS6Q3h/kjsjDhBo90o2fDr+gKKRBiTmA==
-----END CERTIFICATE-----
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
Server certificate
subject=/CN=lonesysadmin.net
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
SSL handshake has read 3260 bytes and written 398 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 3B67BD8D78293E6C2CD87E192316DF2B0DD5B8D8D3E0209DD2A2F2CBE0D8298C
Session-ID-ctx:
Master-Key: BA7C4F7737DA489457285514FA66E935EAD13D4D8DAADA7577917A9B4564120759535FCF76C6616CC96108C375DA015A
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 9b 60 9e 06 36 26 95 27-0a b5 3e ba e2 9f e2 5c ...6&.'..>....\ 0010 - 71 f1 c4 12 2f 73 60 5e-ed 3b 19 fd af 48 51 4d q.../s^.;…HQM
0020 - 85 47 93 5b b4 83 45 ef-04 15 ba 59 85 96 eb c1 .G.[..E….Y….
0030 - 70 da e2 6f c4 f5 99 b5-ed c0 c2 6b 67 73 85 4e p..o…….kgs.N
0040 - 3e f1 6f e2 3c 5c f9 1f-e9 d3 8b c1 96 53 ea b2 >.o.<.……S..
0050 - dd a8 e9 0e 20 5c a5 de-c9 80 cc c6 35 62 c1 51 …. .…..5b.Q
0060 - c0 64 b3 2f ca eb 15 97-2a cd ef 51 8e 5f 21 32 .d./….*..Q.!2 0070 - 4b d9 f9 2e ba ec b1 e5-06 cb dc 57 ab 1d 23 28 K……….W..#( 0080 - 76 41 9c 79 e4 05 23 68-c4 2c 0c f1 46 df 55 01 vA.y..#h.,..F.U. 0090 - 0e 68 d8 83 53 e1 8d 02-18 d4 b0 3d fc a6 03 9a .h..S……=…. 00a0 - 2c 68 88 79 91 4b c9 ba-47 40 b4 aa d3 fb 17 e5 ,h.y.K..G@…… 00b0 - d5 36 f2 45 10 70 dd c4-1e be 69 6a d0 88 e1 a7 .6.E.p….ij…. 00c0 - ac 5f df ef b1 e7 bc be-42 06 8f 8c f3 82 95 5c .……B……\
Start Time: 1543255454Timeout : 300 (sec)
Verify return code: 0 (ok)
DONE

Just prune out everything that isn’t between a “BEGIN CERTIFICATE” and “END CERTIFICATE” line:

-----BEGIN CERTIFICATE-----
MIIFWDCCBECgAwIBAgISA/05rMV6H+0LKP7uo3EE2F5zMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODExMTIwNTEzMTBaFw0x
OTAyMTAwNTEzMTBaMBsxGTAXBgNVBAMTEGxvbmVzeXNhZG1pbi5uZXQwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC22CK5EPHggy7q6qgqiYObGumi4b6X
DV/xKXauS5P2w2zjUFnHO01KRPDQ2owrc4opNRbngqanBI6llWTlBTrMCJSCa8sQ
uE6PEp7Hs94y1xHKKWQ+Lkk8ha99E50plAAU0CY3M7qqJ5Js5Q6L+MZ94H7/4Vvr
c0Ojiu6iXug5YmeNteqE3gzJvQcNoEk8js4HUUuH1FJNm1dLtKFY1/NCut6M39zC
1QOh7YfuGj/DgZD0le24SlBFYgv/2kDwVdPesIktyuV0aPZ+gihLFAjQotCvfhZ1
lQ8/+n9gFlG0vpjVdv8ZurixeyUykVk/Xq9HZLPwaDCubJ2NT+tZoqyXAgMBAAGj
ggJlMIICYTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFMnifCC04NozOYpDuRzN16P
+WuOMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEB
BGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZy8wGwYDVR0RBBQwEoIQbG9uZXN5c2FkbWluLm5ldDBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AHR+2oMx
rTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQe8xWAAABZwaNaRwAAAQDAEgwRgIhAOWY
Xz2ucXa1Src4Ec6cBR6C6TNttfkGGR9mzTGM/SvZAiEA+EgD4cX9Wl+fqctk3iB8
ho/efgo3Urh9BPPQYefb4zQAdQApPFGWVMg5ZbqqUPxYB9S3b79Yeily3KTDDPTl
RUf0eAAAAWcGjWrZAAAEAwBGMEQCIGmIhmJ1x68Z0s32ZTTv6NBD1pzRaSTo92t8
mybQWsnsAiAg+R3O4ruTA/ao1OZfjlg5GG8/+9Z6qtTGLF0bK+cyDTANBgkqhkiG
9w0BAQsFAAOCAQEASsi4FWVLjkLG04BgtnPWyxWoPTFY16MDTLpat6Clf9oQTjb4
KcpNn9tYlLyv9NWNYnQiD351IvDAoPG1Lk4iGdQlo1kFNlrjyozZlQGax7g1XuXX
9OasSVK5aJtIGStR7J2NQsgbLgbY9AajC9BWE7lCPE9AsRIDj96nN3DgbL8hng3d
2G2CBUEYC9+FqnhpqNUXuogLNr9SUm6AODLsRMGoQ0lXwLmjrb++tKuEn55SlHHF
GvDHKB/qCfjLFpByk05J49v2qDBJcxmOkyTHyi9TnDBplWDRCHO99J6qgMorWma3
UavqxAHS6Q3h/kjsjDhBo90o2fDr+gKKRBiTmA==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

And ta-dum! you can paste that into whatever needs it. Some stuff might need it in reversed order, so if it doesn’t work this way just rearrange it.

Don’t forget to use the correct hostnames and ports! If your AD DC is called dc-01.goatrodeo.org and the global catalog is on port 3269 it’d be:

openssl s_client -showcerts -servername dc-01.goatrodeo.org -connect dc-01.goatrodeo.org:3269 < /dev/null

Good luck!

What You Need to Know About Upgrading to an iPhone Xs or Xr

Topic: iPhoneI just got a new iPhone Xs Max. I had an iPhone 6s which I liked a lot, but it’s been a few years and with more travel I thought I’d enjoy having a better device with me. There are a few things that bit me in the duff.

  1. Some two-factor authentication (2FA) apps like Duo or Google Authenticator store their data in the iPhone Secure Enclave, which isn’t backed up to iCloud or via iTunes. That means that when you switch devices (or if you lose your device) you could lose access to your accounts, or it’ll be a serious pain to regain access (which is the point of 2FA). So don’t trade in your old phone until you’ve re-registered everything!

    @funnelfiasco suggested to me that I switch to Authy, which allows backups and multi-device access. @millardjk suggested keeping screenshots of the registration QR codes, which could be a security issue but would help immensely for reregistering between apps. @vcixnv suggested that SAASPASS was a solution for him, instead of Authy. Thanks Ben, Jim, and Britton!

    (I went with Authy by the way, and I used the Google Authenticator features rather than Authy-specific stuff in what is probably a futile attempt at avoiding lock-in).

  2. No button is kind of a pain in the duff. Get the home screen by swiping up from the bottom bar (the “home bar”).

  3. Switch apps by swiping left and right on the home bar, or by swiping up from the bar to the middle of the right side of the screen. Yes, that’s convoluted. I miss the button.

  4. Summon the control center by swiping down and towards the center from the top right (same as with iPads now). You can customize those controls in Settings -> Control Center -> Customize Controls. That might be new in iOS 12, or not, but I hadn’t noticed it.

  5. There was a feature called “Reachability” on the old phones, which will bring the top of the screen down. On my new phone it was off. You can enable it by going into Settings -> General -> Accessibility and flipping it on. Once it’s on you can swipe from just above the home bar down to move the screen down. It’s still a bit kludgy, but hey.

  6. So I tried to shut the phone off, and discovered they moved it from the side button. Much cursing ensued. You now have to hold the side button and one of the volume buttons together for two seconds. It’s also the SOS function so when you do that you’ll have to enter the passcode again. I’m fine with that, I don’t want someone to point it at my face and gain access to my phone.

  7. If you want to take a screenshot it’s the side button and the volume up. If you tap on the screenshot afterwards you can mark it up or delete it, which is nice.

  8. Make sure that cellular data is on. Mine got shut off in the conversion. If your provider allows it make sure you’re using LTE for both voice and data. This is important on CDMA networks because the older modes didn’t allow for simultaneous voice and data. That’s a bummer if you’re using your phone as a hotspot. Settings -> Cellular -> Cellular Data Options. While you’re in there you might check out the WiFi calling options, too.

  9. Face ID is interesting. I think I like it so far, but it took a little to get used to it. My wife and I set our phones up so we can each unlock them without knowing each other’s passcodes (this is 95% being able to see a recipe on the other person’s phone while we’re cooking, and 5% me updating her phone). Instead of registering a fingerprint, Face ID allows for an “alternate appearance” which we just use for the other person. Settings -> Face ID & Passcode -> Add Alternate Appearance.

  10. With Face ID the auto-lock gets set to 30 seconds. Yuck. While I was in Settings -> Display & Brightness I also disabled Raise to Wake. If you just tap the screen it’ll wake up. I also set Settings -> Notifications -> Show Previews to “When Unlocked.” I don’t like others being able to see my incoming communications.

Overall, do I like it? Sure. The Xs Max is large and expensive and it’s taking me some time to adjust to the size (I’m a week into it), but I think I’ll like it overall going forward. I just wish some of this was on the “Welcome to iPhone” setup spiel, and not just Siri and Apple Pay.

CODE Keyboard

“You spent $150 on a keyboard?” – My wife

There are two kinds of people in technology: those with an opinion about their keyboard, and everybody else. I happen to be one of the first.

Buckling Spring image courtesy of Wikipedia.

I grew up using the IBM Model F and M keyboards. They have a spring in the key switches that buckles as you press down. That gives you two things: a prominent clicking sound from the keypress, and solid tactile feedback from the key. You definitely know when that key switch actuated.

Years ago I had to give up my Model M keyboards. They’re built to last but it was getting harder to find working ones, it was getting inconvenient to adapt them to USB from PS/2, and a case of carpal tunnel made it painful to use a keyboard that required a decent amount of force to type. This also pleased my coworkers, who didn’t particularly like the stream of loud clicking when I was in the office. And so I settled on a series of Dell keyboards, mostly because we had some sitting around. The multimedia controls on the newer Dell Business keyboards are nice, and I’ve been using those for a while now.

“Does it do cool things?” – My six year old daughter

In a few weeks I’m not going to have coworkers within 50 feet of me, and my old keyboards are getting a little, well, old. So I thought I’d treat myself to a new keyboard. Over the last couple years I’ve been lurking in the community around keyboards, marveling at the incredible love that people pour into the devices at their fingertips. In particular, Massdrop has a quite the stream of interesting keyboards and customizations, many available for purchase. There are cheaper options there but I don’t like ground-effect lighting for my keyboard enough to spend $500, though.

Turns out you can buy a faithful clone of the IBM Model M from Unicomp, but I think I’m past the mega-clicky stage of my life. I don’t want people to hear all that when I’m on the phone. So after looking around I decided on a 104-key CODE Keyboard, which is a collaboration between Jeff Atwood of Stack Overflow fame and WASD Keyboards. You can choose the switches that are in it so you get exactly what you want for noise, feel, and actuation pressure. The keys have backlighting, which is great. The keyboard weighs a couple pounds, so you can defend your home office with it if you need to, and it has big patches of rubber underneath so it does not move. It’s got a standard USB cable (micro to A), so you can replace it or customize it, and a bunch of routing options underneath. And best of all, it’s simple & clean.

It’s got six DIP switches on the back to customize it if you are a Mac, Windows, or UNIX person (if you’re used to a Sun keyboard that swapped Ctrl and Caps Lock). I flipped the sixth switch so that the keyboard Function key can do the multimedia controls (versus an OS “menu” key). If you want to customize it further you can just order a WASD v2 keyboard and customize it fully, from a variety of languages and layouts to what color each key is. I liked the compromises and the LED backlighting in the CODE model, but I can order new keycaps in the future if I want.

“I AM A BAT. I FLY.” – My three year old son, unfazed by a new keyboard

Best of all, I was looking for a reason to try it out, so I wrote this. It’s definitely a different feel than my old keyboard, but that’s what I wanted. I like it so far. At the beginning here I was doing a lot of double capitalization (WRiting THings Like THis), but 600 words in that seems to have cleared up. I think this keyboard and I might get along just fine.

Now I need to find an amazing mouse to go with it. Thoughts?

Joining VMware

“We changed again, and yet again, and it was now too late and too far to go back, and I went on. And the mists had all solemnly risen now, and the world lay spread before me.” – Pip, Great Expectations

Growing up the son of a firefighter and homemaker, I was fortunate to have been given the opportunity to go to college so many years ago. So in the autumn of the release of Windows 95 I left my childhood home to go to school at the University of Wisconsin – Madison. At four hours by car the UW was far enough away from my parents that they wouldn’t stop in randomly, but it was close enough that I could go home easily. I never really went home, though. Sure, I’d go visit, but my home became Madison, and I dug in. And while my parents helped with my tuition, room & board was solely my responsibility. I got a job, hired at the UW-Madison Help Desk to do phone support for the dial-in modem pool.

Information Technology wasn’t a career path when I was in high school, at least according to the school guidance counselor who told me I was going to be a chemical engineer, and that was that. All engineering students go through the first sets of classes together, though, and along the way I heard about Electrical & Computer Engineering. Took me about 12 seconds to switch. The grass is always greener, it seems, and it didn’t take long for me to figure out that I liked the software side more than hardware. The overlap with Computer Science seemed a natural path.

Fast-forward a few years. I’d been promoted out of the Help Desk. I was running giant AIX systems for our PeopleSoft implementations, and I was wondering what was next in my life. The work I was doing was so much more interesting than school, and it was the path I wanted to be on. I liked the UW, I had lots of friends there, and the people I was working with and for had interesting problems to solve. Above all, it was safe and familiar. My father died in 2001 and that left me adrift and with a case of PTSD, so when the UW offered me a real job, with real pay and real benefits, I signed on.

23 years later I’ve been fortunate to have worked with some of the brightest (and interestingly enough, fastest and strongest) folks around. I’ve been able to reinvent my job a few times, as new technology comes along to reshape the landscape. Landscaping in higher education involves a lot of hard work, overcoming inertia of silos, culture, and incredible fear of change. It requires immense amounts of patience. It has worn on me, as I’d seen my father’s job as a first responder wear on him, turning us into sarcastic, bitter, angry people. I grew more and more like the mythical Sisyphus, destined to roll rocks up hills as punishment for offending self-appointed gods in non-specific ways.

I’ve been thinking about moving on for a while now. I don’t want to turn into my father, and I cannot keep rolling the rock uphill for 20 more years. I’ve talked to a number of friends that have made the leap to vendors, all of which told me, nicely, to shut up and do it. I clearly enjoy technology, but I also enjoy speaking and writing about it to help others understand more. I’ve been active in the VMware community for years. With all of that I’ve been envious of the work the VMware Technical Marketing folks do in all these spaces, getting paid to do the things I basically do as a hobby.

With two small children I’ve been hesitant to take a position with a lot of travel, though, and I’m very fortunate to be in a spot where I could take some time to make sure where I was going is a very good fit. That said, it took almost no time for me to respond when I was asked to consider applying for a position at VMware, in the Cloud & Platform Business Unit’s Technical Marketing group. I am the secret Mike Foley’s been dying to reveal on Twitter, and I’m very excited to work with him, Adam Eckerle, Niels Hagoort (who just joined as well) and all the others that produce such great content and understanding for VMware customers.

I start at VMware in early December and for the first time in a long time I feel again like Pip in that quote above, excited and nervous at the possibilities that lay before me.

Fixing X11 Forwarding Over SSH and with Sudo

X11 forwarding over SSH not working? Not setting $DISPLAY correctly in your shell? Having problems with X11 and sudo? Yeah, me too. Total pain in the duff. Here’s what I do to fix it. I’m thinking about Linux when I write stuff like this but a lot of this has worked on AIX and Solaris, too.

  • Make sure your SSH client supports X11 Forwarding and that it’s turned on. I use SecureCRT but I know it works in PuTTY as well. Once you turn it on in your client & save the settings you will need to reconnect, the forwarding is established with the connection.
  • Ensure xauth and xterm are installed. You need xauth for this to work, and xterm is a lightweight way to troubleshoot this stuff (just run “xterm” at a shell prompt and a window should pop open).
  • If you are using a command-line client, or forwarding across multiple hosts, is X11 forwarding enabled in your ~/.ssh/config file? Add “ForwardAgent yes” and “ForwardX11 yes” to it. You can also force it with “ssh -X user@host” when you connect.
  • Do you have an X Windows server running on your desktop PC? I use Windows on my desktop and I use VcXsrv. Make sure it’s started and running. VcXsrv asks me how I want to run it, I always choose “Multiple windows,” set the display number to -1 to let it choose, and start no client. You can futz with the rest once you know it’s working.
  • Is your $DISPLAY variable being set but you get errors? If so, that’s usually not forwarding, that’s something on your PC. Check your $DISPLAY with “echo $DISPLAY” at a prompt. It should have something in it like “localhost:10.0” or “localhost:13.0” or so. Does your X Windows server software (VcXsrv) have permissions? If so, set them wide open (allow all hosts to connect).
  • On your SSH server do you have “X11Forwarding yes” and “AllowAgentForwarding yes” in sshd_config? If it’s commented out uncomment it and restart the SSH daemon (“service sshd restart” works on a lot of distros).
  • Is your home directory writable? When you log in it’ll need to create an ~/.Xauthority file and if it cannot do that you’ll have problems.
  • Is your ~/.ssh directory writable and correct permissions? It should be owned by your user and chmod 700. Things in it should be chmod 600.
  • Is there an old ~/.Xauthority file sitting there? Try removing it and logging in again.
  • Did you disable IPv6? If you run “sysctl net.ipv6.conf.all.disable_ipv6” and it comes back as 1, or “lsmod | grep ipv6” shows nothing you might have IPv6 disabled. Turns out OpenSSH hates that and has a very passive-aggressive way of showing it. Add “AddressFamily inet” to your sshd_config and restart the daemon. That forces it back to IPv4 only.
  • Are you trying to run something as root using sudo or su? Getting “X11 connection rejected because of wrong authentication?” That gets funky because of permissions with xauth. There are lots of tricky fixes with xauth but I’ve just found copying my .Xauthority file to my target user works great. Then you can “sudo xterm” with impunity. You might try avoiding “sudo su -” as the hyphen wipes your environment out, and along with it your $DISPLAY. Just try “sudo -u targetusername command” instead.
sudo cp ~plankers/.Xauthority ~root/.Xauthority
  • If you’ve gotten this far and you’re still not able to run ‘xterm’ and have it pop a window open I’m surprised. Try SSHing with debugging on, “ssh -v -X user@host” and see if it tells you what’s wrong. Add more “v” to increase the debugging level, like “ssh -vv -X user@host.”
  • What do the logs say when you connect to the server? A lot of times when there’s something wrong it’ll put something in the logs about what it is.
  • Absolute vanilla installs of Linux distributions usually work fine. As a last resort try a VM running a stock installation of something like Ubuntu and see what happens.

Good luck! I hope at least some of this helps.

Fixing Veeam Backup & Replication Proxy Install Errors

Every once in a while I struggle a little to add a new Veeam Backup & Replication hot-add proxy. If you’re like me and seeing proxy install errors maybe some of these will fix you up. This is what worked for me on Windows Server 2016 when I was getting error 0x00000057, “Failed to create persistent connection to ADMIN$” and some other unhelpful messages.

If you’re using a hardened Windows installation all bets are off, since the goal of hardening is to intentionally disrupt remote access. I’d get it running with as close to a stock Windows installation as possible and then work from there if you need to secure things further. There are also ways to manually install the Veeam Transport Service that might be more helpful.

You might want to consider taking a snapshot before this work, so when you discover what fixes the problem you can revert the snapshot and just implement the fix cleanly.

  1. First, try specifying the username as the full “DOMAIN\Username” format when you add it to the Backup & Replication console. Don’t use the “.\username” format and don’t omit the domain part itself. If you are using local accounts you’ll want to specify “SERVERNAME\username” instead, using what the proxy knows as its name. This alone fixes 90% of the issues I’ve seen.
  2. If you aren’t using the Administrator account (and it’s a good idea not to) does the account you want to use have Administrator rights on the proxy VM, and the correct password? I sometimes forget to add the domain service account I created to the local administrators group.
  3. Check to see if you can reach the administrative shares on the proxy VM. Do this from the Backup & Replication main backup server itself by browsing to \\COMPUTERNAME\\Admin$ using the credentials you’re going to use for Veeam. This may mean you need to use “net use” to map it so you can specify a different username. If that works you should see the Windows directory on the remote computer.
  4. Didn’t work? Is the firewall enabled? For troubleshooting try adding an explicit “allow any” rule for all traffic to & from the backup server. If that makes browsing to Admin$ work then make sure you have rules to permit traffic between the proxy and the other proxies, and the proxy and the main backup server. Note that you can test this by just shutting the firewall off, but don’t do that unless you’re protected in some other way (hardware firewall, etc.).
  5. If the firewall is disabled and you still cannot browse can the backup server ping the proxy? Is there another firewall between them that’s denying traffic?
  6. If the firewall is disabled, they can ping each other, and you still cannot browse have you disabled remote UAC on the proxy VM? Open an administrator-level command prompt and run:
    reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\system /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f

    Reboot the proxy VM and try again. At this point you probably can browse to Admin$, and you should take a moment to make sure your firewall is on and everything is secured again. If you still can’t get in I’d look at more fundamental issues, like time synchronization and DNS.

Good luck!

vSphere 6.7 Will Not Run In My Lab: A Parable

CPU Icon“Hey Bob, I tried installing vSphere 6.7 on my lab servers and it doesn’t work right. You tried using it yet? Been beating my head against a wall here.”

“Yeah, I really like it. A lot. Like, resisting the urge to be irresponsible and upgrade everything. What are your lab servers?”

I knew what he was going to say before he said it. “Dell PowerEdge R610s.” I was actually surprised it was that new, and rack-mountable.

“Yeah, you’re out of luck. CPUs before the E3/E5/E7 family didn’t have VT-x extensions in them to make virtualization easy so VMware had to do this thing called binary translation. vSphere 6.5 was the last release that they supported that on because, frankly, it’s slow and everything associated with that technique is getting really old.”

“What the hell? You’d think they’d tell people about that!”

“What, an obscure KB article with absolutely no practical information in it and a reference in the 6.5 release notes to said obscure KB article didn’t catch your eye?” I say, dripping with sarcasm. “I think there was a warning that flashed on the console of affected hardware when you booted, too, but to be honest I only know that because someone mentioned it, I’ve never seen it myself.”

“That’s total crap, like anybody looks at the console. So now what am I going to do? All my gear doesn’t work.”

“One might argue it works just fine. 6.5 will be supported until November of 2021, you could stay on that. You could run 6.7 nested inside 6.5. I know this is a terrifying thought but you could buy some new equipment, too, something that was on a HCL this decade. Given the current generations of CPUs you’d probably be able to cut your VMware licensing in half while doubling your performance. Stick it to the man, or something.”

“Ha! Somehow I doubt my six licenses would attract their attention. I think I’d need four anyhow for vSAN. Maybe I’ll try the nested thing. Thanks man.”

As a side note to my parable here, if you’re thinking about this and have some time before you have to refresh your hardware it’s worth waiting to see how all this Spectre/Meltdown stuff turns out. None of the junk the ferenghis at Intel are shipping today is secure, at any level, especially given the latest wave of vulnerability disclosures. AMD might also turn out to be a good play moving forward, too, if they’re not in exactly the same spot because they blindly copied everything from Intel. The SSD shortages are subsiding so you don’t have to plan 60 or 90 days out anymore. Time will tell, so take some time if you can.

Midnight is a Confusing Choice for Scheduling

Clock IconMidnight is a poor choice for scheduling anything. Midnight belongs to tomorrow. It’s 0000 on the clock, which is the beginning of the next day. That’s not how humans think, though, because tomorrow is after we wake up!

A great example is a statement like “proposals are due by midnight on April 15.”

What you actually said: proposals aren’t welcome after April 14.
What you probably meant: you want the proposals before the date is April 16.

There’s a 24 hour difference there, and if you enforce the deadline accurately people are going to complain because they were all thinking the second thing (before April 16).

Similarly, this is a problem in change notices and customer communications. When you say there’s an outage scheduled for midnight there’s a very good chance someone will misunderstand when that is. Being wrong by an hour in the middle of the night isn’t so bad. Being wrong by 24 hours gets people riled up and you have enough problems as it is.

The second issue with midnight is when folks represent it as 12:00 AM. When you’re moving fast, as many people are, it’s easy to confuse with noon. Even worse when people mess up and write it as 12:00 PM, because in their head midnight is night which is PM. Except, of course, it isn’t.

Last, midnight is a popular time to schedule automated processes. I get it, it’s easy. If you run something at midnight you don’t have to do much processing to separate yesterday from today. The problem is that there’s a ton of stuff already running on the hour, and you’re just piling on. Most people try to avoid shopping when it’s crazy busy, why would you want to run your jobs that way? If you ran your job a bit earlier or later chances are it’ll run faster because you’re not competing with everyone else.

So instead of midnight, what?

 

1. If you care about time then act like you care about time and write your jobs the right way. Or, decide you don’t care about time so much and put a random sleep in them. Jobs don’t have to sleep long, just enough to avoid parts of the hour that end in :00 and :30.

2. Be strict about how you write your times. Write the date in the ISO 8601 format to help avoid global formatting issues (YYYY-MM-DDThh:mmTZD). Mind daylight savings when you add the time zone (-0500 vs -0600, etc.). Don’t be afraid to spell it out in two ways, ISO and how a non-technical reader would want to see it:

“2018-04-05T23:00-07:00 (11 PM Pacific Daylight Time on April 5, 2018).”

3. Don’t schedule things at midnight or noon. Chances are that if you’re scheduling something you could move it to avoid the issue. Deadlines could move to 2200 or 0600 without too much inconvenience, drastically reducing the potential for confusion. Scheduled work could be 2330 (and if you needed to wait until 0000 just adjust the length of the maintenance window). Even if you’re simply telling someone else that something is going to happen, pick a different time that’s clearly inside a specific day.

Time notation drives everybody crazy — look up some of the holy wars around server clocks set to UTC/GMT vs local time. Communication is hard, too, especially conveying technical topics to non-technical people. Let’s be mindful of these tricky spots and work to reduce confusion where we can. That way, instead of ridiculous & angry conversations about definitions of midnight we can have meaningful & clear conversations about the work itself.

No VMware NSX Hardware Gateway Support for Cisco

I find it interesting, as I’m taking my first real steps into the world of VMware NSX, that there is no Cisco equipment supported as a VMware NSX hardware gateway (VTEP). According to the HCL on March 13th, 2018 there is a complete lack of “Cisco” in the “Partner” category:

Cisco Missing from VMware NSX hardware gateway support

I wonder how that works out for Cisco UCS customers.

As I continue to remind vendors, virtualization environments cannot virtualize everything. There are still dependencies on things like DNS, DHCP, NTP, and AD that need a few physical servers. There will also always be a few hosts that can’t be virtualized because of vendor requirements, politics, and/or fear. Any solution for a virtual environment needs to help take care of those systems or it’s not a solution people can use. Beyond that, most people are unwilling to spend precious time and funds on two solutions. The most amazing solution for VM backup, monitoring, or security is useless if you don’t solve my entire problem, which includes the core dependencies I have running as physical hosts.

Folks like Rubrik and Veeam caught on and solved the problem with backup agents. Now we can back up the physical hosts we still have. Extending NSX services, especially security, to the physical systems would help immensely, too. This functionality is “table stakes” now, base functionality customers expect as we design new systems and refresh old ones, but lots of others are missing the boat, too. HPE only has two models of switches listed. Dell only has three. None of them are 25 Gbps. Most of them aren’t certified for recent NSX releases, either.

This seems like a fly in VMware’s NSX ointment. Is it weak demand for NSX that is leading to networking vendors not supporting VXLAN? Or is it terrible networking products that are causing a lack of NSX sales because of their inability to support these features? Whatever it is, this stands as a big opportunity for players like Arista to stand out and eat Cisco, Dell, and HPE’s lunches by being a big and reliable part of the solution, not just another perpetuation of the problem.

How to Troubleshoot Unreliable or Malfunctioning Hardware

CPU IconMy post on Intel X710 NICs being awful has triggered a lot of emotion and commentary from my readers. One of the common questions has been: so I have X710 NICs, what do I do? How do I troubleshoot hardware that isn’t working right?

1. Document how to reproduce the problem and its severity. Is it a management annoyance or does it cause outages & downtime? Is there a reasonable expectation that what you’re trying to do should work the way you expect? That might seem like an odd question, but sometimes other people do the procurement for (and without) us and there are gotchas they didn’t think to ask about.

In my case with the X710s I felt I had a reasonable expectation that the machine would stay up and that standard features like LLDP, which worked fine with other NICs, would work on these.

Being able to reproduce a problem is key. Intermittent issues are really hard to deal with. Get screen shots of the behavior, of the consoles, of the BSODs & PSODs. Get crash dumps if you can.

2. Check the Hardware Compatibility List for the particular OS and hardware you’re trying to use. Make sure it’s on there. If not, you might not have much success in getting support. The HCL might also have clues about driver levels and settings, too.

3. Check the vendor knowledge bases. At the time I was fighting the X710 issues there were no articles about it but now there are, and there are some suggested workarounds.

4. Update the firmware to the latest levels. You should be doing this already as part of your patching process. If you’re having issues your vendor’s support is going to ask you to do this anyhow, so might as well get ahead of it. Do it on the whole machine, not just the malfunctioning component, because sometimes the problem is an interaction somewhere else.

5. Update the driver to the latest levels. The VMware HCL often lists newer drivers you can apply via Update Manager. Try applying one of those. Sometimes a vendor like Intel will supply a newer driver than a server vendor like Dell will qualify. I usually try to stick with what the vendor who sold me the server has for drivers. For Dell & VMware, that often means installing with and/or remediating to the Dell customized ESXi ISO.

6. Update the OS to the latest levels. Again, you should be doing this for security reasons but on the off-chance you aren’t patched up to the latest levels do it and see if the problem persists. Support is going to ask you to do this anyhow. This isn’t saying you need to upgrade to Windows Server 2016 from 2012R2 or anything, just be at current releases of 2012R2. Of course, if you have the opportunity to test against another OS like that it might be a useful data point.

7. Open a support case with your vendor. Let them help you, or at least get it on record that there are problems. Ask for escalation if there isn’t timely progress.

8. Let your sales team know that you are having problems. Ask them how long you have to return the equipment since it isn’t performing correctly. Let them know you opened a support case. Let them know you need escalation because the support folks aren’t resolving your problems. Sales teams want you to be successful, and they absolutely don’t want the equipment returned so they’ll lean on their technical resources to fix your problem.

9. Let your management know that you are having problems. Often, vendors will be having separate conversations with management around business goals and whatnot. Executives need to know that a vendor isn’t delivering on their promises. I guarantee that the vendor isn’t going to bring it up in conversation so you need to. Besides, most executives & managers I know love a way to derail a sales pitch.

This is also very important if this equipment needs to be installed and operational in certain timeframes. Management might need to adjust project timelines, reset customer expectations, or do some damage control. Get ahead of it.

10. Let your purchasing people know that you are having problems. If this is new equipment they might want to get involved before they pay the vendor, or stop payment until this is resolved. Governmental & SLED entities sometimes have other mechanisms of recourse under their vendor contracts which can be very helpful.

11. Don’t be afraid to tell the vendor that their ideas aren’t an acceptable fix. For example, the LLDP problems on X710 cards have a fix in newer drivers, but it’s completely manual, and will not work if your card is partitioned.

If you need the partitions then you’re stuck with no LLDP, which is crap. If you have a large cluster or value your time (and even if you don’t your employer probably does) a time-consuming, hard-to-maintain manual fix is unacceptable, too. You paid a price premium for X710 cards and you expect them to be fully supported & functional in your OS. Frankly, you could have paid less and had a NIC that actually worked as advertised out of the box.

12. Have someone high in your organization start the conversation around returning the equipment. This is basically the nuclear option, but you might have to do it. If you’ve done the other steps here this shouldn’t be a surprise. In my case with the X710s we said “it’s been three months with no resolution, we either need to return this equipment or get replacement NICs.” Because we’d worked through it and offered them a chance to resolve it, and there wasn’t a resolution, Dell did right by us and got us replacement Broadcom NICs. Problem solved.

Finding a way through situations like these is half linear troubleshooting and half good communications. Make sure you are doing both. Good luck!

%d bloggers like this: