Extraneous SSH Public Keys added to Authorized Keys file on Linux VM

Identify

To check whether you have additional keys, review the authorized keys file (vi .ssh/authorized_keys file) to determine whether any additional keys that you didn’t intend to include have been added.

It is safe to manually remove any additional ssh public keys that might have been added. This will not affect the features that are deployed together with the VM. Also, it will not affect your specified SSH key pair for authentication.

If you do not know or cannot differentiate which public keys in the .ssh/authorized_keys file you specified for authentication, follow these steps: 

1. Review your deployment templates:

   1. ssh public keys

   2. ssh keys in cloud-init configuration

2. Retrieve the deployed ssh keys at creation time from inside the VM, if you have sudo/root access. To do this, follow these steps:

   1. Check the cloud-init configuration that was passed in CustomData: sudo cat /var/lib/waagent/ovf-env.xml | grep "<ns1:CustomData>"
      Use the CustomData value and then use base64 decode to get the public keys that you deployed: echo "<customData value>" | base64 -D

   2. Alternatively, check the Instance Meta Data Service (IMDS) to see the ssh public key that was passed in the ssh public key property of the VM Create: curl -H Metadata:true "http://169.254.169.254/metadata/instance/compute/publicKeys?api-version=2018-04-02&format=json"

Remediate

If you have identified additional certificates that you did not intend to deploy to the VM, you can remove these by erasing the corresponding line from the authorized_keys file.

Run the remediation by connecting to the VM interactively, or use either the Custom Script Extension or the RunCommand across multiple VMs.

VMs deployed by using extensions that have protected settings or a managed identity

Use the following script to remove public keys from certificates in which the VM was deployed with extensions or managed identity. This will not remove keys that were specified when you deployed a VM, or if the VM was deployed with Key Vault keys.

#!/bin/bash
set -e

# /var/lib/waagent has *.crt files that include the crt files corresponding to 
# the user provided public keys and one additional .crt file from MSI.
# This script converts the content of the .crt file into the ssh public key and
# remove it from the authorized_keys file
readarray -t CRT_FILES < <(grep -l -E "(Microsoft.ManagedIdentity|Windows Azure)" /var/lib/waagent/*.crt)
for ((i=0; i < ${#CRT_FILES[@]}; i++))
do
    PUBKEY=$(openssl x509 -in "${CRT_FILES[$i]}" -pubkey -noout | ssh-keygen -f /dev/stdin -i -m PKCS8)
    sed -i -e "\@$PUBKEY@d" $HOME/.ssh/authorized_keys
Done 

After the script has run, check the ssh/authorized_keys file to ensure only the known public key(s) are present.

VMs deployed with Key Vault Secrets

To identify whether the key was added when deployed with Key Vault Keys, follow these steps:

1. Get the name of the Key Vault certificate that you deployed by using the VM, review deployment code Az CLI or ARM templates, or run this Az CLI:
     az vm show --resource-group <resourceGroupName> --name <vmName> | grep certificateUrl
   
   The response will show the certificate name:
     "certificateUrl": "https://<keyVaultname>.vault.azure.net/secrets/<certName>/xxxxxxxxxxxxx"

2. Download the certificate:
     az keyvault certificate download --vault-name <keyVaultName> --name <certName> --encoding PEM --file public.pem

3. Extract the public key:
     openssl x509 -in public.pm -pubkey -noout | ssh-keygen -f /dev/stdin -i -m PKCS8

4. Compare the output from the previous step to the remaining certs in the ssh/authorized_keys file.
     vi .ssh/authorized_keys file

Ubuntu 18.04 images

To update a source custom image, you must make the following modifications:

• Edit the following file:
  
  /etc/cloud/cloud.cfg.d/90-azure.cfg

• Add the following code to the end of the file.

datasource:
   Azure:
     agent_command: [service, walinuxagent, start]

RHEL 7.4/7.5/7.6 and CentOS 7.6 images

If you previously created the RHEL/CentOS images by using these steps (or a similar method), you must update the source image from which you created the VMs. The following are additional steps that are required to add to your existing source VM image configuration:

Step 1 

Edit the following file:

/etc/cloud/cloud.cfg.d/91-azure_datasource.cfg 

Add the following lines to the end of the file: 

datasource:
   Azure:
     agent_command: [systemctl, start, waagent, --no-block]

Step 2

Update the Agent configuration as follows:

cp /lib/systemd/system/waagent.service /etc/systemd/system/waagent.service
sed -i 's/After=network-online.target/WantedBy=cloud-init.service\\nAfter=network.service systemd-networkd-wait-online.service/g' /etc/systemd/system/waagent.service
systemctl daemon-reload

cloud-init packages

All cloud-init packages that include a fix are in the process of being updated. Microsoft is researching this problem and will post more information in this article when the information becomes available.