Applicability:
These steps are designed to be run on servers which were previously connected to Azure Arc-Enabled Servers, where connectivity was lost due to incident JLJ3-B88.
If you did not receive a Service Health notification regarding incident JLJ3-B88, these instructions are not relevant. If you have an issue with Azure Arc-Enabled Servers connectivity, please contact support.
To validate locally whether a server is affected, run 'azcmagent show':
❯ azcmagent show
Resource Name : test Resource Group Name : test Resource Namespace : Subscription ID : 11111111-1111-1111-1111-111111111111 Tenant ID : 11111111-1111-1111-1111-111111111111 VM ID : 11111111-1111-1111-1111-111111111111 Correlation ID : 11111111-1111-1111-1111-111111111111 VM UUID : 11111111-1111-1111-1111-111111111111 Location : westeurope Cloud : AzureCloud Agent Version : 1.17.01931.201 Agent Logfile : C:\ProgramData\AzureConnectedMachineAgent\Log\himds.log Agent Status : Disconnected Agent Last Heartbeat : 2022-05-12T11:27:15-07:00 Agent Error Code : Agent Error Details : Agent Error Timestamp : Using HTTPS Proxy : Proxy Bypass List : Cloud Provider : N/A Cloud Metadata Manufacturer : Dell Inc. Model : XPS 8930 MSSQL Server Detected : false Dependent Service Status GC Service (gcarcservice) : running Extension Service (extensionservice) : running Agent Service (himds) : running
Not affected: If the Subscription ID, Tenant ID, Resource Name or Resource Group Name is not set, the machine has not been connected and is not affected by this issue. Onboard it using 'azcmagent connect' if desired. If the Location is not 'westeurope', the server is not affected by this issue. If the Agent Status is 'Connected', the server is not affected by this issue. You should not run these steps on these servers.
Affected: If the Agent Error Details include a message similar to "AADSTS700016: Application with identifier 'xxx' was not found in the directory 'yyy'" then the server is affected by this issue. If the server is in West Europe, and status is Disconnected, but this message is not shown in the error details, it may be affected. You should run these steps on these servers.
Context: Servers affected by issue JLJ3-B88 are unable to communicate with Azure because the server's Managed Identity has been deleted. These steps are designed to create a new managed identity for the same resource, and update the service to use that identity.
Steps: A tool 'azcmrepair' has been developed to simplify the repair steps. This behaves very similarly to the azcmagent.exe distributed as part of the Azure Connected Machine Agent.
Download for Windows: Download azcmrepair.exe from aka.ms/azcmrepairwindows azcmrepair needs to be run in an Administrative shell on the local server. It can run in any directory and can be deleted after use.
Download for Linux: Download azcmrepair from https://aka.ms/azcmrepairlinux , for example with curl -L https://aka.ms/azcmrepairlinux -o ./azcmrepair Mark the file as executable, if required: chmod +x ./azcmrepair Execute it as root, using sudo azcmrepair.
Usage: For interactive use, you can just use: > ./azcmrepair run You will be prompted for credentials to use while authenticating to Azure. You will need the same permissions on the target Azure subscription and resource group that you would need to onboard servers initially. For at-scale use, the tool can use a service principal in the same way as 'azcmagent connect'. This can use the same service principal used for onboarding, or another one if preferred. The service principal needs at least the "Azure Connected Machine Onboarding" role. > ./azcmrepair run --service-principal-id xxx --service-principal-secret yyy No other arguments are required.
Evaluating Success: The tool writes diagnostic log information to the console and azcmagent.log. If the command is successful, you will see "Successfully Onboarded Resource to Azure" in the output. After this, if you run 'azcmagent show', you should see the server's state as 'Connected'. This will also show in the Portal but there may be a few minutes delay before the status is updated in the cloud. The tool will have a 0 exit code in a successful case. If you see "Machine is already connected, no repair needed", the tool believes that the server is connected already. If 'azcmagent show' also shows the server as connected, no further action is required. In case of failure, the tool will return a non-zero exit code and the output will indicate the issue. Please share any issues with Microsoft Support in this case.
Some common error conditions: Error: Exit Code: AZCM0016: Missing Mandatory Parameter or azcmrepair failed because missing 'xyz' in the agentconfig.json Resolution: Your local agentconfig.json is missing or invalid. It's possible the machine had not been connected before. Run azcmagent connect to connect your machine to Azure. You do not need to run the repair tool in this case. Limitations: Any roles assignments granted to the server's managed identity (for example, allowing it to access an Azure resource such as a Keyvault), are tied to the identity and cannot be restored by the azcmrepair utility. When the utility has run, a new managed identity will have been created for the resource. Please grant any role assignments to the new identity Azure Active Directory.
