AWS Lambda functions are widely used in data engineering for ETL (Extract, Transform, Load) processes. They allow you to read data from multiple sources and to apply cleansing and filtering before loading the data into the final destination. One common issue data engineers encounter when working with lambda functions is difficulties in establishing a connection to RedShift’s database that’s in a private Subnet.

In this article, we’ll discover why a Lambda Function fails to access RedShift’s database.

Reason 1: Lambda’s VPC configuration

As we know, the Virtual Private Cloud (VPC) lets you launch AWS resources in a logically isolated virtual network that you define. In our case, the RedShift cluster is launched within a private-facing subnet with no internet access. That is to say, all attempts to access the database from the public internet will eventually fail, including lambda function requests.

> Solution:

Place your lambda function and RedShift cluster within the same VPC’s private subnet.

Note: Ensure this IAM Managed policy AWSLambdaVPCAccessExecutionRole is attached to your lambda’s IAM role.

Reason 2: Misconfiguration of Security Groups

Security group acts as a virtual firewall that controls inbound and outbound traffic for EC2 instances, Lambda functions, and other AWS resources. It is essential to configure security groups correctly to allow communication between the Lambda function and Redshift.

> Solution:

• Configure the security group associated with your Lambda function to allow outbound traffic to the cluster’s Security group.
• Configure the security group associated with your Redshift cluster to allow inbound traffic from the Lambda function.

Reason 3: Lambda’s role lacks IAM Permissions

AWS IAM **(Identity and Access Management) **Roles **are a fundamental part of the **AWS security model. They define a set of permissions that determine what actions and resources an AWS service or user can access. They are commonly used for granting permissions to AWS services, allowing them to access other AWS resources securely without exposing any credentials. This means the IAM role that is assigned to your Lambda function should have sufficient permissions to access the database.

> Solution:

Make sure that your lambda’s execution role has the following policy: AmazonRedshiftDataFullAccess

Reason 4: Inaccessible Database Connection Credentials

AWS Secrets Manager which provides a secure **and **scalable solution for storing and managing secrets such as database credentials, API keys, and other sensitive information. In our case, Lambda and RedShift are both placed in a private subnet, so we need to find a way to allow access to credentials stored in AWS Secret Manager. The most cost-effective option would be to use an AWS Secrets Manager VPC endpoint.

> Solution:

For a Lambda function to access an AW*S secret* from within a VPC you should check the following steps:

• Ensure lambda’s execution role has the following policy: SecretsManagerReadWrite.

• Make sure that the AWS Secrets Manager and the Lambda function are in the same AWS region. Note: Secrets are region-specific and cannot be accessed from a different region.

• Configure the outbound rules of the Lambda function’s security group to allow access to the Secrets Manager service endpoint.

To sum up, a lambda function might fail to connect to RedShift for multiple reasons but we could group them into these two categories: network (VPC, Security Group, Endpoints) and security (IAM permissions). By following the troubleshooting steps outlined in this article, you would be able to identify and address the root cause of the issue.

References:

• AWS Managed policy to access VPC: https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaVPCAccessExecutionRole.html
• AWS Managed policy to access RedShift: https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRedshiftDataFullAccess.html
• Lambda inside a custom VPC: https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-internet
• Connecting to Amazon Redshift using an interface VPC endpoint: https://docs.aws.amazon.com/redshift/latest/mgmt/security-private-link.html