Deploy AWS Amplify Python Lambda from MacOS with Docker

TL;DR

This article introduces how to use Docker to build and deploy AWS Python Lambda from MacOS (or any non-Linux platform).

Overview

AWS Amplify is a tool to help developers set up and manage AWS resources conveniently. AWS Amplify provides an interactive cli calls amplify-cli to create backend resources and frontend webpages. All of the created resources have the CloudFormation configuration files, which enables development teams to migrate it to self-owned packages or pipelines easily. [Read More]

Coming back to this article’s topic, during my recent use of Amplify Python Lambda, I found the following error if I deploy the lambda from my MacOS:

{
  "errorMessage": "Unable to import module 'index': No module named 'regex._regex'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

The reason to the issue is I have added a dependency to the Pipfile: [code link]

[packages]
nltk = "~=3.4"

When deploying from Mac, pipenv automatically fetches the built binary for MacOS platform. Meanwhile, AWS Lambda runs on Linux platforms and thus requires Linux dependencies.

Solve the Issue

To solve this issue, we must make pipenv to retrieve the Linux dependencies and then deploy. There are two ways to achieve the goal:

Install a Linux system with VirtualBox, and deploy the code from there

• Pros: straightforward
• Cons: inefficient, not CI/CD friendly

Using docker to create a virtual environment, and deploy inside the docker

• Pros: fast, scriptable
• Cons: incompatible with amplify-cli

I went for the Docker solution. Although it does not work with amplify-cli, but the solution is fast and every step is in code, which makes it scalable.

To complete the build and deployment workflow, we need to execute the following steps:

1. Setup an Linux environment
   This step could be done with a properly configured Dockerfile.
2. Download the dependencies in the docker container
   We need to run pipenv install inside the docker container.
3. Zip the AWS Lambda and deploy it to AWS
   Using aws lambda update-function-code can complete this step.

Setup Linux environment

To force pipenv to download the Linux dependencies, I choose the following Docker image, which is the same one that AWS Lambda CI/CD uses:

lambci/lambda:build-python3.8

The benefits of this image are:

1. The image provides a Linux environment
2. The image contains aws command, which be used later

The Dockerfile looks like this: [code link]

FROM lambci/lambda:build-python3.8
ENV AWS_DEFAULT_REGION us-west-2
ENV AWS_ACCESS_KEY_ID [YOUR_AWS_ACCESS_KEY_ID]
ENV AWS_SECRET_ACCESS_KEY [YOUR_AWS_SECRET_ACCESS_KEY]
# copy the code base into Docker container
COPY . . 
# this script will be shown below
CMD ./dockerCmd.sh

Prepare the build and deploy script

As described above, we need to conduct the following steps in the Docker container:

build → zip → deploy

Here is a handy script for dockerCmd.sh which will be executed inside the Docker container: [code link]

#!/bin/bash
# 1. build, fetch the dependencies
pipenv install
# 2. zip, prepare the AWS Lambda to upload
cd `pipenv --venv`/lib/python3.8/site-packages/
zip -r /var/task/my-deployment-package.zip .
cd -
zip -r -g my-deployment-package.zip src/
# 3. deploy, using `aws` command to update the remote AWS Lambda
aws lambda update-function-code --function-name amplifypylambda7935db62-dev --zip-file fileb://my-deployment-package.zip

Deploy and test

As the last step, let’s light up the Docker container and test it.

Initiate Docker container and build & deploy:

docker build -t tmp-awslambda . && docker run --rm tmp-awslambda

Test invoking the lambda: [code link]

{
  "statusCode": 200,
  "body": "[\\"The\\", \\"Docker\\", \\"solution\\", \\"works\\", \\"!\\"]"
}

Conclusion

Generally, the deployment is taken care of by CI/CD once the code is pushed into repos. But sometimes developers need to do ad-hoc deployments in test stages, thus the solution in this article can be helpful.

Using Docker is convenient, feel free to share your thoughts in the comments below!

[Github Repo]

If you find this article useful, please follow this account for future updates. Thanks for the support!