Welcome back! Since you know you'll be deploying this model via Docker on Lambda, that dictates how your inference pipeline should be structured.
You need to build a “controller”. What is that exactly? It's just a function that accepts the JSON object that is passed to Lambda and returns the results of your model, again in a JSON payload. Therefore, everything your inference pipeline will do needs to be called inside this function.
In the case of my project, I have a whole codebase of feature engineering functions: mountains of stuff involving semantic embeddings, a bunch of aggregations, regular expressions, and more. I have consolidated them into a FeatureEngineering class, which has several private methods but only one public one, feature_eng. Then, from the JSON that is passed to the model, that method can execute all the necessary steps to get the data from “raw” to “features”. I like to configure this way because it abstracts a lot of complexity from the controller function itself. I can literally just call:
fe = FeatureEngineering(input=json_object)
processed_features = fe.feature_eng()
And I'm off to the races, my features appearing clean and ready to go.
Please note: I have written extensive unit tests on all the internals of this class because while it is nice to write it this way, I still need to be extremely aware of any changes that may occur under the hood. Write your unit tests! If you make a small change, you may not be able to immediately realize that you've broken something in the process until it's already causing problems.
The second half is the inference work and in my case this is a separate class. I've gone for a very similar approach, which only includes a few arguments.
ps = PredictionStage(features=processed_features)
predictions = ps.predict(
feature_file="feature_set.json",
model_file="classifier",
)
The initialization of the class accepts the result of the feature engineering class method, so that the handshake is clearly defined. The prediction method then takes two things: the feature set (a JSON file that lists all the feature names) and the model object, in my case a CatBoost classifier that I've already trained and saved. I'm using the native CatBoost save method, but whatever you use and whatever model algorithm you use is fine. The point is that this method abstracts a bunch of underlying stuff and clearly returns the result. predictions object, which is what my Lambda will give you when it runs.
So to summarize, my “controller” function is essentially this:
def lambda_handler(json_object, _context):fe = FeatureEngineering(input=json_object)
processed_features = fe.feature_eng()
ps = PredictionStage(features=processed_features)
predictions = ps.predict(
feature_file="feature_set.json",
model_file="classifier",
)
return predictions.to_dict("records")
Nothing else! You may want to add some checks for malformed inputs so that if your Lambda gets an empty JSON, a list, or something else strange, you're ready, but that's not necessary. However, make sure your output is in JSON or similar format (here I return a dict).
This is all great, we have a Poetry project with a fully defined environment and all the dependencies, plus the ability to load the modules we created, etc. Good material. But now we need to translate that into a Docker image that we can put on AWS.
Here I show you a skeleton of the docker file for this situation. First, we used AWS to get the correct base image for Lambda. Next, we need to configure the file structure to be used within the Docker image. This may or may not be exactly the same as what you have in your Poetry project; mine isn't, because I have a bunch of extra junk here and there that isn't necessary for the production inference process, including my training code. . I just need to put the inferences into this image, that's all.
The beginning of the docker file
FROM public.ecr.aws/lambda/python:3.9ARG YOUR_ENV
ENV NLTK_DATA=/tmp
ENV HF_HOME=/tmp
In this project, everything you copy will live in a /tmp folder, so if you have packages in your project that will try to save data at any time, you need to direct them to the right place.
You should also make sure that Poetry is installed directly on your Docker image; that's what will make all your carefully selected dependencies work correctly. Here I am setting the version and counting pip to install Poetry before continuing.
ENV YOUR_ENV=${YOUR_ENV} \
POETRY_VERSION=1.7.1
ENV SKIP_HACK=trueRUN pip install "poetry==$POETRY_VERSION"
The next problem is making sure that all the files and folders your project uses locally are correctly added to this new image – the Docker copy sometimes irritatingly flattens directories, so if you build this and start seeing problems “module not found”, check to make sure that is not happening to you. Hint: add RUN ls -R to the dockerfile once everything is copied to see what the directory looks like. You will be able to see those logs in Docker and it could reveal any problems.
Also, make sure you copy everything you need! That includes the Lambda file, your poetry files, your function list file, and your model. All of this will be necessary unless you store them somewhere else, like in S3, and have Lambda download them on the fly. (That's a perfectly reasonable strategy for developing something like this, but not what we're doing today.)
WORKDIR ${LAMBDA_TASK_ROOT}COPY /poetry.lock ${LAMBDA_TASK_ROOT}
COPY /pyproject.toml ${LAMBDA_TASK_ROOT}
COPY /new_package/lambda_dir/lambda_function.py ${LAMBDA_TASK_ROOT}
COPY /new_package/preprocessing ${LAMBDA_TASK_ROOT}/new_package/preprocessing
COPY /new_package/tools ${LAMBDA_TASK_ROOT}/new_package/tools
COPY /new_package/modeling/feature_set.json ${LAMBDA_TASK_ROOT}/new_package
COPY /data/models/classifier ${LAMBDA_TASK_ROOT}/new_package
We're almost done! The last thing you need to do is install your Poetry environment and then configure your driver to run. There are a couple of important flags here, including --no-dev which tells Poetry not to add any development tools you have in your environment, perhaps like pytest or black.
The end of the docker file
RUN poetry config virtualenvs.create false
RUN poetry install --no-devCMD ( "lambda_function.lambda_handler" )
That's it, you now have your docker file! Now is the time to build it.
- Make sure Docker is installed and running on your computer. This may take a second but it won't be too difficult.
- Go to the directory where your dockerfile is, which should be the top level of your project, and run
docker build .Let Docker do its thing and then when it has completed the build it will stop returning messages. You can see in the Docker application console if it was created successfully. - Go back to the terminal and run
docker image lsand you will see the new image you just created and it will have an ID number attached to it. - From the terminal once again, run
docker run -p 9000:8080 IMAGE ID NUMBERwith your ID number from step 3 completed. Now your Docker image will start running! - Open a new terminal (Docker is attached to your old window, just leave it there) and you can pass something to your Lambda, which is now running through Docker. I personally like to put my entries in a JSON file, like
lambda_cases.jsonand run them like this:
curl -d @lambda_cases.json http://localhost:9000/2015-03-31/functions/function/invocations
If the output in the terminal is the model's predictions, then you're ready to rock. If not, check the errors and see what could be wrong. Chances are you'll have to do some debugging and troubleshooting before everything runs smoothly, but that's part of the process.
The next stage will largely depend on your organization's setup and I'm not a Devops expert so I'll have to be a bit vague. Our system uses AWS Elastic Container Registry (ECR) to store the created Docker image and Lambda accesses it from there.
When you are completely satisfied with the Docker image from the previous step, you will need to build it once again, using the following format. The first flag indicates the platform you are using for Lambda. (Put a bookmark on that, it will come back later.) The element after the -t flag is the path where your AWS ECR images go; Fill in your correct account number, region and project name.
docker build . --platform=linux/arm64 -t accountnumber.dkr.ecr.us-east-1.amazonaws.com/your_lambda_project:latest
After this, you need to authenticate to an Amazon ECR registry in your terminal, probably using the command aws ecr get-login-password and using the appropriate flags.
Finally, you can push your new Docker image to ECR:
docker push accountnumber.dkr.ecr.us-east-1.amazonaws.com/your_lambda_project:latest
If you've authenticated successfully, this should only take a moment.
There's one more step before you're ready to get started: configuring Lambda in the AWS UI. Sign in to your AWS account and search for the “Lambda” product.
Open the left menu and find “Features”.
This is where you will go to find your specific project. If you haven't set up Lambda yet, hit “Create Function” and follow the instructions to create a new function based on your container image.
If you've already created a feature, find that one. From there, all you need to do is hit “Deploy new image.” Regardless of whether it's a completely new feature or just a new image, make sure you select the platform that matches what you did in your Docker build. (Remember that pin?)
The last task, and the reason I have continued to explain up to this point, is to test your image in the real Lambda environment. This can lead to bugs that you didn't find in your local tests! Go to the Test tab and create a new test by entering a JSON body that reflects what your model will see in production. Run the test and make sure your model does as expected.
If it works, then you made it! You have implemented your model. Congratulations!
However, there are a number of potential hiccups that can appear here. But don't panic if you have a mistake! There are solutions.
- If your Lambda runs out of memory, go to the Settings tab and increase the memory.
- If the image didn't work because it is too large (10 GB is the maximum), go back to the Docker build stage and try reducing the content size. Don't package extremely large files if the model can handle them. In the worst case, you may need to save your model to S3 and have the function load it.
- If you're having trouble navigating AWS, you're not the first. Check with your IT or Devops team for help. Don't make a mistake that will cost your company a lot of money!
- If you have another problem not mentioned, please post a comment and I will do my best to advise you.
Good luck, happy modeling!
