Extraneous SSH Public Keys added to Authorized Keys file on Linux VM

Applies to: Virtual Machine running Linux

Summary


In addition to letting users provide their own SSH keypairs for authentication, the Microsoft Azure platform relies on SSH keypairs to enable some features that are added to the virtual machine (VM) at deployment time. We recently discovered that, in some limited scenarios, the public keys from these Azure Platform certificates could be unexpectedly added to the .ssh/authorized_keys file. These scenarios are scoped only to a situation in which the VM is provisioned by using cloud-init and the user selects additional Azure features that rely on certificates, such as a system-managed service identity.

This unexpected behavior occurs because of a change in the provisioning logic of specific operating systems. These are systems that use cloud-init and that inadvertently install the public key from all certificates that are available to the VM into ssh-authorized keys file during VM creation 

To learn more, go to CVE-2019-0816.

More information


Scenario details

The cloud-init logic that is mentioned in the "Summary" section is currently known to exist in Azure images for Ubuntu 18.04 in addition to the Public Preview RHEL 7.4/7.5/7.6 and CentOS 7.4 cloud-init images. It may also exist in custom images using these operating systems. 
 
If you enable one of the following features while you provision one of the Linux images, you may see additional, unexpected keys in an .ssh/authorized_keys file, such as any of the following: 

  • Managed identity
  • Extensions with protected settings
  • Deploy a VM with Key Vault Keys in a VM

Identify and remediate existing VMs

Identify

To check whether you have additional keys, review the authorized keys file (vi .ssh/authorized_keys file) to determine whether any additional keys that you didn’t intend to include have been added.

It is safe to manually remove any additional ssh public keys that might have been added. This will not affect the features that are deployed together with the VM. Also, it will not affect your specified SSH key pair for authentication.

If you do not know or cannot differentiate which public keys in the .ssh/authorized_keys file you specified for authentication, follow these steps: 

  1. Review your deployment templates:
    1. ssh public keys
    2. ssh keys in cloud-init configuration
  2. Retrieve the deployed ssh keys at creation time from inside the VM, if you have sudo/root access. To do this, follow these steps:
    1. Check the cloud-init configuration that was passed in CustomData:
      sudo cat /var/lib/waagent/ovf-env.xml | grep "<ns1:CustomData>"

      Use the CustomData value and then use base64 decode to get the public keys that you deployed:
      echo "<customData value>" | base64 -D
    2. Alternatively, check the Instance Meta Data Service (IMDS) to see the ssh public key that was passed in the ssh public key property of the VM Create:
      curl -H Metadata:true "http://169.254.169.254/metadata/instance/compute/publicKeys?api-version=2018-04-02&format=json"

Remediate

If you have identified additional certificates that you did not intend to deploy to the VM, you can remove these by erasing the corresponding line from the authorized_keys file.

Run the remediation by connecting to the VM interactively, or use either the Custom Script Extension or the RunCommand across multiple VMs.

VMs deployed by using extensions that have protected settings or a managed identity

Use the following script to remove public keys from certificates in which the VM was deployed with extensions or managed identity. This will not remove keys that were specified when you deployed a VM, or if the VM was deployed with Key Vault keys.

 

#!/bin/bashset -e# /var/lib/waagent has *.crt files that include the crt files corresponding to # the user provided public keys and one additional .crt file from MSI.# This script converts the content of the .crt file into the ssh public key and# remove it from the authorized_keys filereadarray -t CRT_FILES < <(grep -l -E "(Microsoft.ManagedIdentity|Windows Azure)" /var/lib/waagent/*.crt)for ((i=0; i < ${#CRT_FILES[@]}; i++))do    PUBKEY=$(openssl x509 -in "${CRT_FILES[$i]}" -pubkey -noout | ssh-keygen -f /dev/stdin -i -m PKCS8)    sed -i -e "\@$PUBKEY@d" $HOME/.ssh/authorized_keysDone 

After the script has run, check the ssh/authorized_keys file to ensure only the known public key(s) are present.

VMs deployed with Key Vault Secrets

To identify whether the key was added when deployed with Key Vault Keys, follow these steps:

  1. Get the name of the Key Vault certificate that you deployed by using the VM, review deployment code Az CLI or ARM templates, or run this Az CLI:
     
    az vm show --resource-group <resourceGroupName> --name <vmName> | grep certificateUrl


    The response will show the certificate name:
     
    "certificateUrl": "https://<keyVaultname>.vault.azure.net/secrets/<certName>/xxxxxxxxxxxxx"
  2. Download the certificate:
     
    az keyvault certificate download --vault-name <keyVaultName> --name <certName> --encoding PEM --file public.pem
  3. Extract the public key:
     
    openssl x509 -in public.pm -pubkey -noout | ssh-keygen -f /dev/stdin -i -m PKCS8
  4. Compare the output from the previous step to the remaining certs in the ssh/authorized_keys file.
     
    vi .ssh/authorized_keys file

Resolution


Azure MarketPlace Images

Fixes have been applied to cloud-init in the identified Azure images:

  • Canonical:UbuntuServer:18.04-LTS:18.04.201902190
  • Canonical:UbuntuServer:18.10-DAILY:18.10.201903200
  • RedHat:RHEL:7-RAW-CI:7.6.2019030421
  • OpenLogic:CentOS-CI:7-CI:7.6.20190306

Custom Images

If you are using custom images that are provisioned by cloud-init for the known operating systems, you will have to update your source custom image.

Ubuntu 18.04 images

To update a source custom image, you must make the following modifications:

  • Edit the following file:

    /etc/cloud/cloud.cfg.d/90-azure.cfg
  • Add the following code to the end of the file.
datasource:   Azure:     agent_command: [service, walinuxagent, start]

RHEL 7.4/7.5/7.6 and CentOS 7.6 images

If you previously created the RHEL/CentOS images by using these steps (or a similar method), you must update the source image from which you created the VMs. The following are additional steps that are required to add to your existing source VM image configuration:

Step 1 

Edit the following file:

/etc/cloud/cloud.cfg.d/91-azure_datasource.cfg 

Add the following lines to the end of the file: 

datasource:   Azure:     agent_command: [systemctl, start, waagent, --no-block]

Step 2

Update the Agent configuration as follows:

cp /lib/systemd/system/waagent.service /etc/systemd/system/waagent.servicesed -i 's/After=network-online.target/WantedBy=cloud-init.service\\nAfter=network.service systemd-networkd-wait-online.service/g' /etc/systemd/system/waagent.servicesystemctl daemon-reload

cloud-init packages

All cloud-init packages that include a fix are in the process of being updated. Microsoft is researching this problem and will post more information in this article when the information becomes available. 

Frequently asked questions


Q1: Has Microsoft gained access to my VM?

A1: Encryption keys are used to manage identities and extensions are not designed for access by Microsoft employees. We have processes in place to monitor, log, and prevent this kind of access. As a safety precaution, we are reviewing all logs to make sure that no customer keys were inappropriately accessed. For certificates that are stored in Key Vault which are referenced in a VM or VMSS deployment, Microsoft employees do not have any access to the secrets.

Q2: Are all cloud-init deployed OSs and versions affected?

A2: No. We have seen this extraneous key issue occur only in the identified operating systems. We have not seen it occur in older versions of Ubuntu. This is because those systems use a different mechanism to set public keys.

Q3: Do the affected cloud-init packages for the Linux distributions include the fix?

A3: We are working on adding the fixes to the packages for the affected versions and we will update this article when that work is completed.

Q4: Will Microsoft automatically update any affected VMs? 

A4: No. Microsoft will not  change the contents of VMs that you have already provisioned.

Q5: Our security policies prevent me from operating a Linux VM with extraneous content in authorized_keys. What can I do about this today?

A5: You can remove the offending line safely from the authorized_keys file. This won’t affect either the SSH keypair that you created and control or the managed identity. You can do this manually or by running a custom script or custom command across any affected fleet.