Troubleshooter: Bad gateway (502) errors in Azure Application Gateway

Applies to: Virtual Network

What does this guide do?

Helps you troubleshoot bad gateway (502) errors that occur in Azure Application Gateway.

Who is it for?

System administrators who use Application Gateway to manage traffic to web applications.

How does it work?

Provides a checklist and sequence of steps to help you identify the issue and reach a resolution.

Estimated time of completion:

15-30 minutes.

After you configure an application gateway, you receive the following errors message:

After you configure an application gateway, you receive the following errors message:

Follow these steps to check whether the back-end pool is empty:

  1. In Azure portal, select All resources, and then select the application gateway that has the problem. 
  2. On the Application gateway blade, select Backend pools.
  3. Check the Targets value. If the value is 0 (zero), that means that the back-end pool is empty.
     
Is the back-end pool empty?

If the back-end pool is empty, and there is no server to forward the request, the application gateway returns an "HTTP 502" error message to the client.

To resolve this issue, follow these steps to add back-end servers to back-end pool:

  1. In Azure portal, select All resources, and then select the application gateway.
  2. On the left menu, select Backend pools.
  3. Select the back-end pool.
  4. Under Targets, select a back-end server type.

    Note The server can be of one of the following types:
    • IP Address/FQDN
    • Virtual machine
    • Virtual machine scale set
    • App service
  5. Select the targeted virtual machine or app service, or type the IP address or FQDN.
  6. Select Save.
     
Did this resolve your issue?

Check whether all the back-end servers are unhealthy.

  1. All resources, and then select the application gateway.
  2. On the Application gateway blade, select the Rule. Identify the HTTP settings and Backend pool values that are tied to the listener by the rule.
  3. On the Application gateway blade, select Backend Health.
  4. Check whether the status on all servers is Unhealthy for the HTTP settings and back-end pool that you identified in step 2.
  5. From the Details field, record the detailed error message that you see. This will be used for further troubleshooting.

     

Are the all back-end servers unhealthy?

If at least one server is healthy, try to increase the request time-out, and then check whether the problem is resolved. To do this, follow these steps:

  1. In Azure portal, select All resources, and then select the application gateway.
  2. On the Application gateway blade, select the HTTP settings.
  3. Select the HTTP setting you created. In the Request Timeout (seconds) box, enter a higher value, such as 120. Or, enter a value that is greater than the number of seconds that your server takes to return the response to every request.
  4. Click Save.

Note To determine how long the server takes to respond, check the Access log. For each request, the timeTaken value (in milliseconds) indicates the whole processing of the request from the first byte that is received by the application gateway to the last byte that is sent out to the client.


Did this resolve your issue?

If you have both basic rules and path-based rules, make sure that the path-based rule is set to the higher priority. To change the priority of rules, follow these steps:

  1. In Azure portal, select All resources, and then select the Application gateway.
  2. On the Application gateway blade, select Rules.
  3. Check whether there is a basic type rule that is listed above the multi-site listener rules. If there is, delete the basic type rule, and then create a rule that has the basic listener. The new rule is put at the bottom of the list. In this manner, multi-site rules are prioritized.

Did this resolve your issue?
Select one of the following error messages that you see in the back-end health status display:

To determine whether a custom probe is configured, follow these steps:

  1. In Azure portal, select All resources, and then select the application gateway.
  2. In the the left menu, select HTTP settings, and then select the HTTP setting that you created.
  3. If the Use custom probe check box is selected, you are using a custom probe.

Is there a custom probe configured in the application gateway?

In the application gateway that has problem, select Health probes. Check the custom probe settings based on the error that you received in the Details field in the Backend health section.

Error

Actions

Cannot connect to server

A TCP session could not be established. Check the port in HTTP Settings, and verify that you can connect to the server on the port or. Also check whether any network security group or user-defined routing is affecting the traffic.

Probe status code mismatch: Received 401

Check whether the back-end server requires authentication. Application gateway probes can’t pass credentials for authentication at this point. Either allow "HTTP 401" in a probe status code match or probe to a path where the server does not require authentication.

Probe status code mismatch: Received 403

Check whether access to the path is allowed on the back-end server.

Probe status code mismatch: Received 404

Check the hostname path if it is accessible on the back-end server. Change the hostname or path parameter to an accessible value.

Probe status code mismatch: Received 405

Check whether the server allows the HTTP GET method.


Did this resolve your issue?

Check whether your server is accessible for default probe parameters:

  • Default probe will be on <protocol>://127.0.0.1:<port>/ and accepts status codes 200-399.
  • The protocol and port are inherited from the HTTP settings section in the application gateway instance.
  • If your server is not accessible on localhost, configure a custom probe by using the appropriate hostname and protocol, and then associate it with the back-end HTTP settings that you are using.
  • If the back-end server responds by sending a different status code that you want to be allowed (such as 401), include that code in the probe matching conditions.

Did this resolve your issue?

If you receive the “Backend Server Certificate needs to be whitelisted in Application Gateway” SSL error message, follow these steps to resolve the issue:

  1. Access the server as specified in the back-end pool. If an IP address or FQDN is specified in the back-end pool, access the server by using https://<ip-address>/ or https://<FQDN>/, and then check the certificate from the web browser.
  2. Go to the Application gateway blade, select HTTTP settings, and then verify that this same certificate has been uploaded in the application gateway for whitelisting. If you find a mismatch, export the certificate’s public key to a base 64-encoded .cer file, and then upload the same key to HTTP settings.

    For more information, see Create certificates for whitelisting backend with Azure Application Gateway.

Note If the back-end server is configured to have SNI (Server Name Indication), you must use FQDN in the back-end pool. This should match the binding in the back-end server in the case of Application Gateway v1 SKU. By contrast, in v2 SKU, you can configure the SNI FQDN in the “Override Host name” section of the HTTP settings. Otherwise, turn off SNI and configure a fall-back certificate.


Did this resolve your issue?

If the probe fails because of an HTTPS probe connection error, follow these steps:

  1. Select All resources in Azure portal, and then select the application gateway.
  2. In the Application gateway blade, select Listeners In.
  3. Check the SSL Policy of your application gateway, and then make sure that the correct TLS version and cipher suites are selected
  4. Verify that your back-end server supports the same TLS version and cipher suites by identifying which TLS version and cipher suites are currently used in the connection. To do this, take a network packet capture on the back-end server by using Wireshark or Network Monitor, and then check the TLS Client and Server Hello packets. For example:
    250	4.045574	10.171.40.80	40.126.18.38	TLSv1.2	268	Client Hello254	4.075988	40.126.18.38	10.171.40.80	TLSv1.2	92	Server Hello, Certificate, Server Key Exchange, Server Hello Done

     

Did this resolve your issue?

Congratulations! Your issue is resolved.

If your issue persists, please contact Azure Support for further assistance.