KB at your finger tips

This is one stop global knowledge base where you can learn about all the products, solutions and support features.

How do I troubleshoot errors with SSL certificates that are generated by API Gateway?

Last updated: 2022-12-15

I'm experiencing issues with self-signed and expired SSL certificates installed on my backend system. How to I fix these errors?

Short description

When Amazon API Gateway performs an SSL handshake with the backend, API Gateway expects the backend to provide certificates that are obtained from trusted issuers. API Gateway expects the certificates to be valid, and not expired. API Gateway also expects the chain of trust to be intact. This means that API Gateway expects the certificate to contain a root certificate authority (CA), intermediate CAs, and the parent certificate details. With this information, API Gateway can complete certificate validation by going through the chain of certificates.

Resolution

Test HTTP proxy integration

To familiarize yourself with HTTP proxy integrations, test bad SSL certificates from the API Gateway console. Use the external website badssl.com that provides bad SSL certificates for testing.

1. Create a resource named "/selfsigned" with a GET method. Then, configure an HTTP proxy integration with the URL https://self-signed.badssl.com/.

From the API Gateway console, test the API. You receive the following error:

Thu Dec 15 16:05:05 UTC 2022 : Sending request to https://self-signed.badssl.com/
Thu Dec 15 16:05:05 UTC 2022 : Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

2. Create a resource named "/expiredcert" with a GET method. Then, configure an HTTP proxy integration with the URL https://expired.badssl.com/.

From the API Gateway console, test the API. You receive the following error:

Thu Dec 15 16:06:02 UTC 2022 : Sending request to https://expired.badssl.com/
Thu Dec 15 16:06:02 UTC 2022 : Execution failed due to configuration error: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

3. Create a resource named "/untrustedRootCA" with a GET method. Configure an HTTP proxy integration with the URL https://untrusted-root.badssl.com/.

From the API Gateway console, test the API. You receive the following error:

Thu Dec 15 16:06:28 UTC 2022 : Sending request to https://untrusted-root.badssl.com/
Thu Dec 15 16:06:28 UTC 2022 : Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

VPC link integration

With VPC link integration, API Gateway performs certificate validation with the next hop that performs TLS termination.

When a Network Load Balancer has a TLS listener, the Network Load Balancer performs a TLS termination and creates another connection to the target. The certificate attached to the Network Load Balancer must meet all the requirements. A Network Load Balancer doesn't perform certificate validation during the SSL handshake with the target. The Network Load Balancer accepts expired or self-signed certificates that are installed on the target instances. The Network Load Balancer and the target groups are bound within a VPC and communications are secure. If the Network Load Balancer is using a TCP listener, the TLS handshake happens end-to-end. In these cases, the backend application must comply with the SSL requirements.

API Gateway supports Server Name Indication (SNI) during an SSL handshake over a VPC link integration.

If the backend Network Load Balancer has a self-signed or private certificate that hasn't been issued by a CA, you receive the following error:

Execution failed due to configuration error: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The workaround for the execution failed error is to set insecureSkipVerification to true in the integration's tlsConfig object:

aws api gateway update-integration --rest-api-id abcde --resource-id abcd --http-method GET --patch-operations "op='replace',path='/tlsConfig/insecureSkipVerification',value= False"

Related information

Generate and configure an SSL certificate for backend authentication

API Gateway-supported certificate authorities for HTTP and HTTP proxy integrations

Target groups for your Network Load Balancers

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Use a third party client certificate with mutual TLS for API Gateway APIs

How can I use a client certificate issued by a third party when configuring mutual TLS authentication for API Gateway APIs?

Last updated: 2022-12-16

I want to use a client certificate issued by a third party when configuring mutual Transport Layer Security (TLS) authentication for Amazon API Gateway APIs.

Resolution

To use a third-party signed client certificate for API Gateway with TLS authentication, follow these steps:

Create a truststore to use the third-party signed certificate.
Create a custom domain name and API mapping for your API Gateway API.
Create a DNS record to map the custom domain name to API Gateway.
Disable the default endpoint.
Test the API.

Note:

If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.
Mutual TLS isn't supported for private APIs.

Create a truststore to use the third-party signed certificate

1. Generate a RootCA. The RootCA is used to issue the client certificate.

openssl genrsa -out RootCA.key 4096

openssl req -new -x509 -days 3650 -key RootCA.key -out RootCA.pem

2. Fill in the required fields, and enter domain name.

3. Create a client certificate private key, certificate signing request (CSR), and client certificate.

openssl genrsa -out my_client.key 2048

openssl req -new -key my_client.key -out my_client.csr

4. Sign the client certificate using the CA that you created previously.

openssl x509 -req -in my_client.csr -CA RootCA.pem -CAkey RootCA.key -set_serial 01 -out my_client.pem -days 3650 -sha256

5. Create an Amazon Simple Storage Service (Amazon S3) bucket to store the truststore.pem file.

6. Upload the RootCA.pem to the Amazon S3 bucket truststore.

7. (Optional) Create a certificate chain file from the intermediate CA certificate to Root CA.

Cat Intermediate.pem ca.pem >ca-chain.pem.

For more information, see Configuring your truststore.

Create a custom domain name and API mapping for your API Gateway API

Note: Only certificates with a key size of 2048 bits are visible with API Gateway. The certificate size can't exceed 2048 bits.

1. Open the API Gateway console, and then choose Custom domain names .

2. For Domain names , enter your domain name, and then choose Create .

3. For Domain name , enter your domain name.

4. For Minimum TLS version , choose TLS 1.2 .

5. Enable Mutual TLS authentication .

6. For Truststore URI , enter the Amazon S3 URI that you created earlier.

7. For Endpoint configuration , choose Regional .

8. For Certificate type , choose your certificate type, and then choose Create domain name .

9. Choose your custom domain name, and then choose Configure API mappings .

10. Choose Add new mapping .

11. Choose the API , Stage , and Path for the mapping, and then choose Save .

For more information, see How can I set up a custom domain name for my API Gateway API?

Create a DNS record to map the custom domain name to API Gateway

Create a DNS record to point the custom domain name to the API Gateway Regional domain name. This allows the traffic that's bound to the custom domain name to be routed to the API's Regional hostname. The DNS record can be the CNAME or "A Alias" type.

For more information, see Configuring Route 53 to route traffic to an API Gateway endpoint.

Disable the default endpoint

By default, clients can invoke APIs using the execute-api endpoint that API Gateway generates for your API. Disable the default execute-api endpoint so that clients can access your API using only a custom domain name.

For HTTP APIs, see Disabling the default endpoint for an HTTP API.

For REST APIs, see Disabling the default endpoint for a REST API.

Test the API

Test the API with the --key and --cert parameters to send the client certificate as part of the request.

$ curl -v https://{YourCustomDomainName}/{resource} --key my_client.key --cert my_client.pem

Note: The private key my_client.key and the client certificate my_client.pem must be included for a successful request.

Related information

How can I troubleshoot certificate chain and self-signed certificate issues for Amazon API Gateway with custom domains and mutual TLS enabled?

Introducing mutual TLS authentication for Amazon API Gateway

How do I troubleshoot HTTP 403 Forbidden errors from an API Gateway custom domain name that requires mutual TLS?

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Read article

Troubleshoot 504 errors from an API Gateway REST API

How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?

Last updated: 2022-12-14

I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?

Resolution

When your REST API requests return an HTTP 504 error status code, you must run various checks.

Check for IntegrationLatency spikes

Check if there's an IntegrationLatency spike in API Gateway by reviewing the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging.

For more information, see Set up Amazon CloudWatch API logging using the API Gateway console.

An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. Check the Lambda function's Duration performance metric to confirm this.

For more information, see Working with Lambda function metrics.

Review requests with CloudWatch Logs Insights

Use Amazon CloudWatch Logs Insights to review requests that resulted in 504 errors. To review the requests, on the CloudWatch console, in the navigation pane, choose Logs , Log Insights . Select your API Gateway log group. Then, set the relative time with one of the following queries:

parse @message '(*) *' as reqId, message
| filter message like /Method completed with status: \d\d\d/
| parse message 'Method completed with status: *' as status
| filter status = 504
| sort @timestamp desc
| limit 20

-or-

fields @timestamp, @message
| filter message like /Method completed with status: 504/
| sort @timestamp desc
| limit 20

Instrument X-Ray traces

If 504 errors keep happening continuously, determine where the Lambda function is spending its time. Instrument AWS X-Ray traces for Lambda functions based on the Lambda function runtime.

For Python:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()

For Node.js:

const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))

Note: You need to build a new deployment package after instrumenting X-Ray traces on a Lambda function.

Implement API retries

When a 504 error occurs and the request isn't found in Lambda, implement API retries on the client. The error might have resulted from a temporary network failure in API Gateway.

Check the Lambda function configuration

Make sure that your Lambda function has only API Gateway event-specific processing logic. This way, the Lambda function takes less time to run and can keep up with incoming events.

REST APIs have a default maximum integration timeout of 29 seconds. Therefore, make sure that the Lambda function's running duration is under 29 seconds.

If you have a use case where the application is time sensitive, see Set up asynchronous invocation of the backend Lambda function.

Related information

API Gateway dimensions and metrics

Setting up CloudWatch logging for a REST API in API Gateway

View API Gateway log events in the CloudWatch console

Tracing user requests to REST APIs using X-Ray

Using Lambda with X-Ray

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Read article

Invoke a Lambda function asynchronously from API Gateway

How can I invoke a Lambda function asynchronously from my Amazon API Gateway API?

Last updated: 2022-12-12

I want to invoke an AWS Lambda function asynchronously instead of synchronously for my Amazon API Gateway API.

Resolution

REST APIs

In Lambda non-proxy integration, the backend Lambda function is invoked synchronously by default. You can configure the Lambda function for a Lambda non-proxy integration to be invoked asynchronously by specifying 'Event' as the Lambda invocation type.

1. Open the API Gateway console, choose APIs , and then choose your REST API.

2. In Resources , choose GET , and then choose Integration Request .

3. In Integration type , choose Lambda Function .

4. Expand HTTP Headers , and then choose Add header .

5. For Name , enter X-Amz-Invocation-Type .

6. For Mapped from , enter 'Event' .

7. Redeploy the REST API.

To invoke the Lambda function with the option for either asynchronous or synchronous, add an InvocationType header.

1. Open the API Gateway console, choose APIs , and then choose your REST API.

2. In Resources , choose GET , and then choose Method Request .

3. In Request Validator , choose the edit icon, choose the dropdown list, and then choose Validate query string parameters and headers .

4. Choose the update icon to save changes.

5. Expand HTTP Headers , and then choose Add header .

6. For Name , enter InvocationType , and then choose Required .

7. In Integration Request , Expand HTTP Headers , and then choose Add header .

8. For Name , enter X-Amz-Invocation-Type .

9. For Mapped from , enter method.request.header.InvocationType .

10. Redeploy the REST API.

Clients can include the InvocationType: Event header in API requests for asynchronous invocations or InvocationType: RequestResponse for synchronous invocations.

For more information, see Set up asynchronous invocation of the backend Lambda function.

HTTP APIs

HTTP APIs only support proxy integrations for Lambda. You can't set the X-Amz-Invocation-Type header in the API Gateway integration for HTTP APIs. You can use two Lambda functions with one acting as proxy.

Example configuration:

HTTP API --> Invoke Lambda1 synchronously --> Invoke Lambda2 asynchronously

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Read article

Troubleshoot high latency in API Gateway requests

How do I troubleshoot high latency in my API Gateway requests that are integrated with Lambda?

Last updated: 2022-12-08

Response times are slow when I make requests to an Amazon API Gateway API that’s integrated with an AWS Lambda function. How do I determine the cause of high latency?

Resolution

High latency must be addressed when an API endpoint that’s integrated with a Lambda function takes too long to send responses to a client. Review the metrics for API Gateway to identify the section of the request/response flow that’s causing high latency. After you determine the cause of high latency, you can work to reduce the delays.

Filter CloudWatch metrics to review latency metrics on the API

To identify the section of the request/response flow that’s causing high latency, first perform the following steps:

Observe the latency of the client after sending a request to the API.
After you note the overall latency, open the Amazon CloudWatch console. In the left navigation pane, choose Metrics , All metrics . In the metrics search box, enter APIGateway . From the search results, choose API Gateway , ApiId .
In the list of APIs, filter for the specified API by using the API ID or the API name. After filtering, check the IntegrationLatency and Latency check boxes.
Note: The API ID and API name are available from the API Gateway console.
Open the Graphed metrics tab. For Statistic , choose Maximum . For Period , choose 1 minute . Above the graph, select the Custom time period. Choose the time frame during which the client experienced high latency.
Review both the IntegrationLatency and Latency metrics. Note the values and timestamps when these metrics have high values. The values can explain the cause for high latency.

Compare metrics to identify the cause of high latency

Continue to review the metrics related to the request/response flow to find the cause of high latency:

Compare the API Gateway Latency metric to the overall latency value observed at the client.
For example, an API has a Latency metric with a Maximum value that’s approximately equal to the Max Latency value at the client. These values suggest that the maximum delay in the request/response flow is the time taken by API Gateway to process requests. API Gateway processing time includes the time taken to send requests to Lambda, wait for responses from Lambda, and send responses to a client.
Compare the IntegrationLatency metric with the Latency metric for the API.
For example, the IntegrationLatency metric is approximately equal to the Latency metric. These values indicate that latency at the API is primarily caused by backend requests sent to Lambda that are taking longer to respond. The IntegrationLatency metric includes the time between API Gateway sending a request and API Gateway receiving a response from the backend
When the IntegrationLatency metric is low when compared to the Latency metric for the API, the backend response times are low. In this scenario, it takes longer to process the API requests or responses.
For example, mapping templates configured in the API or an API Gateway Lambda authorizer both might create delays.
When the Latency metric for the API is much lower than the latency observed at the client, the route might be causing delays. Review the route between the client and API Gateway to confirm whether there are any intermediate endpoints that are adding delays.
For example, private VPN connections or proxies might create delays.

View Lambda metrics to identify the cause of high IntegrationLatency

Finally, focus on the Lambda metrics related to the request/response flow to find the cause of high IntegrationLatency :

Check the Lambda function Duration metric to confirm whether the Lambda function’s execution time is longer. If the Lambda function’s execution time has increased, review the CloudWatch log to find the section of code that’s causing high latency. By default, Lambda functions log the START, END, and REPORT statements in CloudWatch logs. Add custom log statements at each logical section of the Lambda function code to get verbose CloudWatch logs.
If the Duration metric didn't change during the period of high latency at the client, determine if the initialization time increased. Initialization time in a Lambda function is the time taken to set up the execution environment to process a request. Requests that come from API Gateway might require a new environment for processing. This is set up through Lambda. Typically, the code that’s present outside the Lambda function handler runs during the initialization time. Code that takes longer to complete can cause delays for the overall response times to the client.
Note : Initialization time is known as INIT or cold start.
Confirm whether there’s any increase in the initialization time’s Duration by verifying the report statements in the Lambda function logs. Initialization time that's high for some requests can cause an increase in the IntegrationLatency metric for API Gateway.

Related information

Working with metrics for HTTP APIs

Amazon API Gateway dimensions and metrics

Monitoring WebSocket API execution with CloudWatch metrics

Viewing metrics on the CloudWatch console

Lambda runtime environment lifecycle

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Read article

Troubleshoot API Gateway logs

How can I use Amazon API Gateway logs for troubleshooting?

Last updated: 2022-12-07

I want to use Amazon API Gateway logs for troubleshooting API issues.

Short description

To troubleshoot an API Gateway REST API or WebSocket API, turn on execution logging and access logging using Amazon CloudWatch Logs. Make sure that you have attached all the required permissions to the API Gateway AWS Identity and Access Management (IAM) role. For instructions, see How do I turn on CloudWatch Logs for troubleshooting my API Gateway REST API or WebSocket API?

Note: HTTP APIs currently support access logging only, and logging setup is different for these APIs. For more information, see Configuring logging for an HTTP API.

Resolution

After you turn on logging, API Gateway automatically creates a CloudWatch log group named API-Gateway-Execution-Logs/{rest-api-id}/{stage-name} . For more information, see CloudWatch log formats for API Gateway.

AWS Lambda authorizers

If you have Lambda authorizers turned on for your API, the logs indicate if the request is authorized and include the authorizer and request IDs.

Example authorized logs:

Starting authorizer: 62mgrc for request: 3323ffe5-6ddf-45c8-896a-b45ec2284cc0
Successfully completed authorizer execution

Example unauthorized logs:

Starting authorizer: 62mgrc for request: fd443056-60d0-425f-874e-e93e5d61351e
Unauthorized request: fd443056-60d0-425f-874e-e93e5d61351e

Usage plans

After the Lambda authorizer logs, API Gateway verifies the usage plan for the API request. If you haven't configured a usage plan, then API requests aren't counted for throttling limits.

Example usage plan logs:

Verifying Usage Plan for request: 3323ffe5-6ddf-45c8-896a-b45ec2284cc0. API Key: API Stage: <api_id>/<stage>
API Key authorized because method 'GET /' does not require API Key. Request will not contribute to throttle or quota limits
Usage Plan check succeeded for API Key and API Stage <api_id>/<stage>

If you have configured APIs with API keys but the X-API-Key header is missing, then the logs return an error similar to the following:

API Key not authorized because method 'GET /' requires API Key and API 
Key is not associated with a Usage Plan for API Stage 
<api_id>/<stage>: API Key was required but not present

If you added the correct API key in the header but the request is throttled, then the logs return an error similar to the following:

API Key **********************************FGQes7 exceeded throttle limit for API Stage <api_id>/<stage>: Key throttle limit exceeded for Usage Plan ID iwuzkt. Limit: 5.00 Burst: 0
Method completed with status: 429

Request log types

Note: All request log types require turning on Log full requests/responses data in the execution logs.

Method request logs

Method request logs includes detailed information received by an API request including the request path, query parameter, request header, and request body.

Endpoint request logs

Endpoint request logs include detailed information received by an API request including the URI, request headers, and request body.

The data included in endpoint requests are sent to your backend integration for further processing.

Endpoint response logs

Endpoint response logs include the data returned by the backend integration including the status code, headers body, and Integration latency.

Method response logs

Method response logs includes the headers and body sent to the client after the transformation.

Important: API Gateway limits log events to 1024 bytes. Log events larger than 1024 bytes, such as request and response bodies, are truncated by API Gateway before submission to CloudWatch logs.

Related information

How can I troubleshoot missing CloudWatch logs for API Gateway REST APIs?

Setting up CloudWatch logging for a REST API in API Gateway

Monitoring REST API execution with Amazon CloudWatch metrics

Monitoring WebSocket API execution with CloudWatch metrics

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Read article

Welcome to Knowledge Base!

KB at your finger tips

How do I troubleshoot errors with SSL certificates that are generated by API Gateway?

Short description

Resolution

Test HTTP proxy integration

VPC link integration

Related information

How can I use a client certificate issued by a third party when configuring mutual TLS authentication for API Gateway APIs?

Resolution

Create a truststore to use the third-party signed certificate

Create a custom domain name and API mapping for your API Gateway API

Create a DNS record to map the custom domain name to API Gateway

Disable the default endpoint

Test the API

Related information

How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?

Resolution

Check for IntegrationLatency spikes

Review requests with CloudWatch Logs Insights

Instrument X-Ray traces

Implement API retries

Check the Lambda function configuration

Related information

How can I invoke a Lambda function asynchronously from my Amazon API Gateway API?

Resolution

REST APIs

HTTP APIs

How do I troubleshoot high latency in my API Gateway requests that are integrated with Lambda?

Resolution

Filter CloudWatch metrics to review latency metrics on the API

Compare metrics to identify the cause of high latency

View Lambda metrics to identify the cause of high IntegrationLatency

Related information

How can I use Amazon API Gateway logs for troubleshooting?

Short description

Resolution

AWS Lambda authorizers

Usage plans

Request log types

Related information

Still Thinking? Give us a try!

Still Thinking?
Give us a try!